Can one use the SDK call to receive events when Kinesis stream changes number of shards?

0

Hello;

Using the SubscribeToShard SDK call, one can receive events via HTTP when records are added to the shard. And other SDK calls allow one to determine what shards are in a Kinesis stream.

Is there a way to receive events via HTTP when a shard is added to or removed from a stream, so that a program can know to add or remove a consumer for that shard? Or is it necessary to regularly call the ListShards API to determine what shards are in the stream?

Thank you.

  • Hello,

    Using ListShards is one way of subscribing to new shards after resharding operations take place and you could regularly check with the API if new shards are available to do so.

    One thing to note is that the SubscribeToShard API also returns the list of the child shards of the current shard when the end of the current shard is reached, which means that you are able to see the newly split or merged shard ID's which you can then make use of to Subscribe to them.

    If event.ContinuationSequenceNumber returns null, it indicates that a shard split or merge has occurred that involved this shard. This shard is now in a CLOSED state, and you have read all available data records from this shard. In this scenario, per example above, you can use event.childShards in your code logic to subscribe to the new shards.

    For more information you are welcome to have a look at the complete SDK examples that can be found here: https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/kinesis/src/main/java/com/example/kinesis/KinesisStreamEx.java

    You can also reference the following documentation: https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html

asked a year ago183 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions