DynamoDB Streams - Are shards ordered in the stream description?


I’m aware that the items in the stream are ordered. My principle question then is about whether or not the shards are ordered in the stream description. If they are, I can always just grab the latest shard and process records from there; if not, I would need to keep track of the shards already processed in order to get only the new shard from the next iteration when I grab the stream description—I.e., after the current shard is closed and I have to quit out of the getRecords loop for that shard.

Further Context:
I have already complete a KCL implementation, but am looking at doing a lower-level implementation, as that looks to be more suited to our particular needs. Ultimately, I just need to get the latest record from the table (broadcast amongst all of the service nodes). That said, my quick question is as follows: is the shard list returned from DescribeStreamResult.getShards() an ordered list? (I presume this follows the general describe stream API, but have yet to see anything indicating whether or not this list is ordered.)

asked 7 years ago48 views
1 Answer
Accepted Answer

The DynamoDB streams api is "intentionally" similar to Kinesis streams (from the documentation here). This also states that: Using the Kinesis Adapter is the recommended way to consume Streams from DynamoDB. The API documentation for the Kinesis DescribeStream API states that:

There are no guarantees about the chronological order shards returned in DescribeStream results. If you want to process shards in chronological order, use ParentShardId to track lineage to the oldest shard.

The DynamoDB DescribeStream API contains the parentSharedId as well as SequenceNumberRange.

answered 7 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions