DynamoDB Streams: Iterator Age spikes despite healthy Lambda - shard rebalancing issue?
For context, the setup is: A service saves entities in the store, this triggers DynamoDB change data capture (CDC) which triggers a lambda function. The DynamoDB table is configured with PAY_PER_REQUEST billing mode, which I have read may involve shard rebalancing.
The issue here is at the moment the lambda is experiencing frequent Iterator Age spikes (30-57s) while:
- Lambda duration remains normal (~700ms average, ~7s max)
- No throttling observed
- 100% success rate (or close to it)
- No errors in processing
- Concurrent executions working as expected (peaks at ~40)
Question: Is there anything we can do to reduce these iterator age spikes, particularly around shard rebalancing behavior?
- Newest
- Most votes
- Most comments
This happens intermittently as DynamoDB streams rotates shards every 4 hours, which in turn causes Lambda to re-discover the shard graph which can lead to spikes in latency, as you've observed.
This blog post discusses in more detail: https://aws.amazon.com/blogs/database/build-scalable-event-driven-architectures-with-amazon-dynamodb-and-aws-lambda/
This is something that the DynamoDB and Lambda team are aware of, and plan to reduce, however, I have no ETA when this improvement will be available.
Relevant content
asked 4 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 2 years ago
Since this answer is ~a year old, any update on whether this has been improved or fixed? And is there a recommended way to reduce the IteratorAge spikes?
This has reduced greatly, since DynamoDB launched ShardFiltering