Kinesis streams for low-latency apps


A customer is interested in re-factoring their credit-card processing app, and they want to use "streams" to glue the various parts together. They are considering either Kinesis or Kafka. Besides the fact that Kinesis is not PCI compliance (client side encryption), they are concerned about the 5 read TPS limit per shard. That seems somewhat low for a real-time app.

The customer only has 100-200 ms time budget to process the transaction. While I know that Kinesis can pull multiple records per read, this implies a potential 200ms gap between reads. While not a lot, this may be a show-stopper for my customer. Is there any way to make Kinesis more responsive (higher read TPS), or is this gap a non-issue in terms of "real-world" latency?!

asked 7 years ago369 views
1 Answer
Accepted Answer

Even though the throughput limits are not modifiable, customer can achieve higher throughput by sizing their Kinesis streams with enough number of shards and choose partition key wisely to distribute incoming records evenly across the shards.

There is a calculator available on Kinesis console, which can be used to calculate the number of shards needed based on the record rate and size.

Customer also should tune their Kinesis producer and consumer code to optimize the performance using KCL/KPL

answered 7 years ago
profile picture
reviewed 14 days ago
profile picture
reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions