Retrieving DynamoDB stream records on AWS Lambda function

Question

Lambda function needs to retrieve all stream records within a certain number of hours from DynamoDB streams. What is the  best way to achieve this? The lambda needs to be invoked by a third party at regular intervals to return the incremental data. So in that way it seems not directly compatible with using DDB streams as event source unless perhaps it was sent to sqs queue for decoupling.

It needs to have no data loss with tens of thousands of records a day. Is using aws-kcl worker a reliable way to achieve this? Using aws-sdk on one lambda instance is problematic due some shards taking too long to query as calls to getrecords with nextsharditerator can take a lot of time to process.

Uri Segev · Answer

Retrieving the records from the stream yourslef is challenging. You will need to be able to read from the different shards (and their number might change). You need to checkpoint your location. You need to handle function timeouts (if it takes longer than 15 minutes to read all records from the previous invocation) and more.

Given the Lambda payload size limit (6 MB), I do not think that the function just reads the records and returns them to the 3rd party, unless the 3rd party knows how to invoke the function multiple times until it returns all the data.

You also mentioned that you can't loose data. There may be cases in which teh 3rd party will not invoke the function in time (maybe the 3rd party is down, network issues, etc.). As the retention in DDB stream is 24 hours, you may loose data.

For the above reasons I would recommend that you use the DDB streams event source for Lambda, and let the Lambda process the records, or store them in an interim location (S3, SQS, etc.). Then the 3rd party can reads the data from there.

Retrieving DynamoDB stream records on AWS Lambda function

Add your answer

Relevant content