- Newest
- Most votes
- Most comments
When you change the number of shards, presumably with the update-shard-count api, Kinesis is taking care of merging and splitting individual shards to get to your desired number of shards. Records in kinesis are immutable, outside of aging off the stream, and need to be able to be read in order. So you end up with a lineage of shards. A parent shard that contained all records within a given hash key range, and then two child shards that split that range and take new records going forward. When those child shards are created the parent shard doesn't go away, the records stay in place in that parent shard and the child shards start taking new records. This also works in reverse when you merge two shards.
When you do the describeStream call each shard in the shards list will list a ParentShardId if it has one. This allows you to contruct a map of the lineage to start reading in the correct place. This is also some of the complexity that is handled for you if you're using the KCL or a Lambda consumer
Relevant content
- asked a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 24 days ago
- AWS OFFICIALUpdated 5 months ago
- AWS OFFICIALUpdated 2 years ago