Kinesis Stream Best Practices and Calculations

I came across this Kinesis Shard Calculator which takes inputs as number of Consumers, Message Size, Throughput, Consumption speed and then recommends the number of Shards. Try it !!

https://comcastsamples.github.io/KinesisShardCalculator/

Amazon Kinesis Data Streams has the following stream and shard quotas and limits.

• There is no upper quota on the number of streams you can have in an account.

• The default shard quota is 500 shards per data stream for the following AWS regions: US East (N. Virginia), US West (Oregon), and Europe (Ireland). For all other regions, the default shard quota is 200 shards per data stream.

• To request the shards per data stream quota increase, follow the procedure outlined in Requesting a Quota Increase.

• A single shard can ingest up to 1 MB of data per second (including partition keys) or 1,000 records per second for writes. Similarly, if you scale your stream to 5,000 shards, the stream can ingest up to 5 GB per second or 5 million records per second. If you need more ingest capacity, you can easily scale up the number of shards in the stream using the AWS Management Console or the UpdateShardCount API.

• The maximum size of the data payload of a record before base64-encoding is up to 1 MB.

• GetRecords can retrieve up to 10 MB of data per call from a single shard, and up to 10,000 records per call. Each call to GetRecords is counted as one read transaction.

• Each shard can support up to five read transactions per second. Each read transaction can provide up to 10,000 records with an upper quota of 10 MB per transaction.

• Each shard can support up to a maximum total data read rate of 2 MB per second via GetRecords. If a call to GetRecords returns 10 MB, subsequent calls made within the next 5 seconds throw an exception.