Incremental archive from DynamoDB, Glue Bookmarking?

Question

My customer is currently archiving DynamoDB data to their data lake with a Glue job and the DynamoDB connector. They've asked if instead of a full load each day they can do an incremental.

I see Glue supports bookmarking which looks like what I need but it only supports S3 (and some JDBC) sources for this.

Is there a way to achieve this?

Tasio · Accepted Answer

Hi,

Have they considered using DynamoDB Streams + Lambda + Kinesis Data Firehose to continuously feed the data lake in S3?

This [blog post][1] describes something similar for expired items with a TTL.

[1]: https://aws.amazon.com/blogs/database/automatically-archive-items-to-s3-using-dynamodb-time-to-live-with-aws-lambda-and-amazon-kinesis-firehose/

Incremental archive from DynamoDB, Glue Bookmarking?

Add your answer

Relevant content