Logic for determining which Kinesis shard a partition key goes to

0

Hi there,

I am currently working with Kinesis shards and I would like to make sure that the hashing function is identical to another message bus I am using. What this entails is I need to write a custom hash function for that other message bus.
I have read the following documentation for Kinesis: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
https://docs.aws.amazon.com/cli/latest/reference/kinesis/describe-stream.html

My understanding from reading the documents is that each shard out of n shards starting from 0 will have a range of keys 2^128/n and each event's partition key will be hashed out to a 128-bit integer via md5 hashing. Eg.

For n partitions (starting from 0):
Partition0 key range:
Starting Hash: 0
Ending Hash: (2^128 / n) - 1
Partition1 key range:
Starting Hash: (2^128 / n)
Ending Hash: 2*(2^128 / n) - 1
...
...
Partition(n-2) key range:
Starting Hash: (n-2)(2^128 / n)
Ending Hash: (n-1)
(2^128 / n) - 1
Partition(n-1) key range:
Starting Hash: (n-1)*(2^128 / n)
Ending Hash: 2^128 - 1

Is there any official documentation or references around this method? I want to make sure my understanding above is correct so I can ensure that the custom method I implement is identical to the Kinesis partitioning mechanism.

Edited by: VictorL85 on Apr 23, 2020 5:12 PM

asked 4 years ago1250 views
2 Answers
0

Your understanding is correct, and you can validate the key ranges yourself via the DescribeStream or ListShards output. However, one thing to note is that the number of shards/partitions in Kinesis is not immutable, and Kinesis does not enforce equal sized shards either. In other words, be mindful of the following cases

  • UpdateShardCount can be used to change the number of shards, which can cause the key ranges to change

  • SplitShard or MergeShards can be used to split/merge key ranges, so shards can be responsible for unequal key ranges.

Edited by: rohitnataws on Apr 25, 2020 11:09 AM

Edited by: rohitnataws on Apr 25, 2020 11:12 AM

AWS
answered 4 years ago
0

Thank you very much!

Good call on the shard splitting. I read through that as well.

Cheers,
Victor

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions