Hello - if I understand correctly, you have a table with composite primary key (partition key and sort key), so the content is a large number of item collections - each in sorted order. You want to find the last item in each collection, and if it has a particular attribute you want to update that last record to remove the deprecated attribute.
You should be able to do this in a controlled manner using Scan - to avoid competing with throughput you use for regular production load, use the Limit parameter to Scan in small pages. Loop through this and introduce a suitable delay in each iteration of the loop. In your loop, identify the first item you see for a new item collection - the item right before it is the last in the prior item collection. Check to see if it has the deprecated attribute, and if it does, issue an UpdateItem call where the update expression applies the REMOVE function for the deprecated attribute - you can condition this on the item still existing (just to make sure some other thread has not already deleted it).
Conceptually I think this process should work - I would encourage you to test outside of production first of course.
Thanks for the inputs PeteNaylor-AWS. Can you also give an idea where we would need to run this? Should we go for Lambdas?
I'm not really an expert on compute choices sorry, but I would imagine this can be solved in an EC2 instance, a container, or perhaps broken out across many Lambda invocations. Depending on the use case, my go-to would be to construct a tool (perhaps a Python script since that's familiar to me) and run it on an EC2 instance. There are many options and approaches available - the best choice in tooling might just be what you are most comfortable with.
Thanks PeteNaylor-AWS, We discussed as a team and thought of EC2 running python script too.
Thanks a lot for your inputs
Dynamo db streams together with EC2 instance.asked a month ago
dynamo DB pagination last pageasked 5 days ago
Can we do write-object-if-not-exist in S3 similar to conditional put in Dynamo DB?asked 9 months ago
Dynamo scan not finding records of time in unix.Accepted Answerasked 4 months ago
How to create dynamo db stream event filter for a field from array of objects?asked 6 months ago
Dynamo db cost usageAccepted Answerasked 5 months ago
How to scan the server filesasked 6 months ago
Can lambda know the specific attribute which changed in the dynamo db table update, when a lambda is triggered by dynamo db table update item??asked 8 months ago
Need AWS Glue to store bad records/ records with error when reading Mongo db data to a S3 path and process the rest of the data.asked 7 months ago
Scan entire dynamo db and update records based on a conditionasked 2 years ago