I'm not sure why you state 98% of data as that only adds complexity to the process. Would 100% of the data which exceeds 90days not meet your requirements?
timestamp attribute seems to meet the TTL criteria, its an epcoh time in seconds and of type number. For that reason, you can simply enable TTL on your table and DynamoDB will delete all the items exceeding 90days free of charge.
Otherwise this is the other option:
As you main priority seems to be keeping the cost as low as possible, it is best if you include rate limiting on the client side to ensure the process doesn't over consume capacity which would increase costs:
- Scan the table, Parallel Scan is preferable for large tables. Use a
FilterExpressionto only return items which exceed 90 days.
- Rate limit the Scan so that it does not consume large amounts of capacity, this can be done by adding
sleep()to your logic
- Delete the items returned in each page of the
- Checkpoint the
LastEvaluatedKeylocally for each page you delete, you want to be able to pick up where you left off in case the application crashes for any reason.
Going forward it is advisable to add a TTL attribute to your tables so DynamoDB will delete the items for free on your behalf.
I'll point out it would be 50x more cost effective to write the 2% of the data to a new table rather than to delete 98% of the data. After migration just delete the original table (zero cost). If you want to accept writes while the scan/copy migration process is happening, you can bifurcate all writes to both tables until the migration process finishes.
How does data retrieval work with a DynamoDB Glue Catalog table?Accepted Answerasked a month ago
DynamoDB: Time frame to avoid stale readasked 7 months ago
Incremental Data Capture from DynamoDB to S3Accepted Answerasked 2 years ago
Needs to clean-up stale data from DynamoDB table which is older than 90 days by keeping cost as low as possibleasked 16 days ago
RDS/DynamoDB/DMS questionsAccepted Answerasked 3 years ago
Read/Write data from existing dynamoDB table using AWS Amplify with IOS appasked 21 days ago
Import Existing DynamoDB table schema to NoSQL workbench's Data modelasked 3 years ago
DynamoDB On Demand Provisioned Cost OptimizationAccepted Answerasked a year ago
What is the best way to import data to an existing DynamoDB table?asked a month ago
How to Delete all Dynamodb records except last 1 year (dynamodb table does not have a TTL attribute set)asked 24 days ago