Help improve AWS Support Official channel in re:Post and share your experience - complete a quick three-question survey to earn a re:Post badge!
Unexpected number of requests from Dynamo using Dynamo Streams
Hello, everyone.
I was doing some testing with AWS Dynamo Stream with a very basic setup: Dynamo table that has Stream "on" sends New and Old Image for an AWS Lambda that simply prints what is received. My Dynamo table has about 5 records that I was playing to insert, delete and update.
Then I got really surprised when I checked the usage of my Free Tier and noticed that I've consumed more than 1,000,000 requests of Dynamo. Also, when I have the trigger for the lambda activated, I get a constant value of ReturnedBytes (avg) and ReturnedRecordsCount (sample count) even when I'm not performing any changes on the Dynamo table.
- Is the 1,000,000 requests expected for a 5 records Dynamo table that I was just playing around?
- Should the ReturnedRecordsCount and/or ReturnedBytes stay "always on" even when there is no new insertion/updating/deleting?
- Is it possible that the unexpected request amount is connected to the "always on" behavior of the stream?
- All of this is expected or am I doing something very wrong?
- Newest
- Most votes
- Most comments
My assumption here is that you have created a recursive loop between DynamoDB and Lambda. If you are writing back to the same DynamoDB table from the Lambda function, you will continue to invoke the Lambda based on the stream event.
The behavior you're experiencing with DynamoDB Streams is not typical for a small table with only 5 records and occasional modifications. Let's address your questions:
-
The 1,000,000 requests for a 5-record DynamoDB table is not expected during normal testing. This high number of requests suggests there might be an issue with your setup or how the Lambda function is interacting with the stream.
-
ReturnedRecordsCount and ReturnedBytes should not stay "always on" when there are no new insertions, updates, or deletions. Typically, these metrics should only show activity when there are actual changes to the table.
-
The unexpected request amount could be related to the "always on" behavior you're observing. This suggests that your Lambda function might be continuously polling the stream, even when there are no new events.
-
This behavior is not expected and indicates that something is likely misconfigured in your setup.
Here are some potential causes and solutions to investigate:
-
Check your Lambda function's event source mapping configuration. Ensure that the batch size and batch window are set appropriately. A very small batch size or window could cause frequent polling.
-
Verify that your Lambda function is not stuck in a loop or continuously invoking itself. Make sure it's properly handling the stream records and not causing unnecessary reads or writes.
-
Examine the CloudWatch logs for your Lambda function to see if it's being invoked repeatedly without new data.
-
Check if you have multiple consumers (Lambda functions or other services) reading from the same stream, which could multiply the number of read requests.
-
Ensure that your Lambda function is not experiencing errors that cause it to retry processing the same records multiple times.
-
Verify that you haven't accidentally set up a circular dependency where changes made by the Lambda trigger new stream events.
To resolve this issue, you should review your Lambda function code, check its configuration, and ensure that it's only processing new events from the stream. You may also want to temporarily disable the trigger and monitor the metrics to see if the unexpected behavior stops.
If the issue persists after these checks, you should consider reaching out to AWS Support for further assistance, as this behavior is not normal for a small-scale DynamoDB Streams setup.
Sources
DynamoDb stream batch size | AWS re:Post
Troubleshoot missing records in DynamoDB streams | AWS re:Post
How Channel Corporation modernized their architecture with Amazon DynamoDB, Part 2: Streams | AWS Database Blog
Relevant content
- asked 3 years ago
- asked a year ago
AWS OFFICIALUpdated 2 years ago
Yep. I've done this, but I wasn't sure because I couldn't find any metrics showing such high numbers. I couldn't find any metrics related to the AWS DynamoDB Stream requests, and I checked the number of Lambda invocations and it's nowhere near a million. However, after I stopped writing to the same table, the number of requests is now well-behaved.