DynamoDB Streams + Bulk TTL Delete

Question

I have a DynamoDB table with ~12 billion rows (all less than 1KB) that have been collected over many years, all populated with a TTL column that set it 1 year from creation date. I'm looking to enable TTL while using DynamoDB stream for sending data to firehose for archival, though I'm concerned that when I enable TTL the lambda function I created won't be able to handle the sheer number of rows that would initially be deleted, so I've got a few questions about this to hopefully alleviate my concerns:

1) How fast will records be deleted with TTL, assuming all records need to be deleted, will this obey write throughput for the table and/or the stream? 
2) If TTL does have a limit of some sort (i.e. 40,000 WCU), would this then take multiple days to process all the records, or would it stop after a certain number of records?
3) Assuming I use event-source-mapping with the max 10,000 item batch, will this be quick enough for the rate the items are deleted?
4) What would happen if event-source-mapping is unable to empty the stream in time before the items are expired, are they just lost forever?

* I am aware of the 5 year limit on TTL, but for this example assume everything is within 5 years for a worst case scenario

Leeroy Hannigan · Accepted Answer

1. TTL uses a background process which in turn only uses spare (excess) capacity, this means it will never exceed your table or stream capacity.
2. TTL works on a best effort basis, if there is spare capacity it will delete the items, if there is no spare capacity, it will wait. It can take hours/days sometimes to delete items.
3. Batch size won't matter too much, but you can monitor your IteratorAge to make sure it stays withing 24 hours, otherwise you run the risk of losing events.
4. If you fail to capture an event after its been in the stream 24 hours, the event is lost, yes.

DynamoDB Streams + Bulk TTL Delete

Add your answer

Relevant content