Logic for determining which Kinesis shard a partition key goes to

0

Hi there,

I am currently working with Kinesis shards and I would like to make sure that the hashing function is identical to another message bus I am using. What this entails is I need to write a custom hash function for that other message bus.
I have read the following documentation for Kinesis: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
https://docs.aws.amazon.com/cli/latest/reference/kinesis/describe-stream.html

My understanding from reading the documents is that each shard out of n shards starting from 0 will have a range of keys 2^128/n and each event's partition key will be hashed out to a 128-bit integer via md5 hashing. Eg.

For n partitions (starting from 0):
Partition0 key range:
Starting Hash: 0
Ending Hash: (2^128 / n) - 1
Partition1 key range:
Starting Hash: (2^128 / n)
Ending Hash: 2*(2^128 / n) - 1
...
...
Partition(n-2) key range:
Starting Hash: (n-2)(2^128 / n)
Ending Hash: (n-1)
(2^128 / n) - 1
Partition(n-1) key range:
Starting Hash: (n-1)*(2^128 / n)
Ending Hash: 2^128 - 1

Is there any official documentation or references around this method? I want to make sure my understanding above is correct so I can ensure that the custom method I implement is identical to the Kinesis partitioning mechanism.

Edited by: VictorL85 on Apr 23, 2020 5:12 PM

gefragt vor 4 Jahren1273 Aufrufe
2 Antworten
0

Your understanding is correct, and you can validate the key ranges yourself via the DescribeStream or ListShards output. However, one thing to note is that the number of shards/partitions in Kinesis is not immutable, and Kinesis does not enforce equal sized shards either. In other words, be mindful of the following cases

  • UpdateShardCount can be used to change the number of shards, which can cause the key ranges to change

  • SplitShard or MergeShards can be used to split/merge key ranges, so shards can be responsible for unequal key ranges.

Edited by: rohitnataws on Apr 25, 2020 11:09 AM

Edited by: rohitnataws on Apr 25, 2020 11:12 AM

AWS
beantwortet vor 4 Jahren
0

Thank you very much!

Good call on the shard splitting. I read through that as well.

Cheers,
Victor

beantwortet vor 4 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen