Skip to content

At Realtime, efficiently comparing records in DynamoDB with record in RDS (PostgreSQL)

0

What is the best solution to ensure that the data in RDS is the latest, considering the data is being sent to RDS via middleware that fetches the data from DynamoDB? The data volume is high (~3000K in 10 min in high traffic duration) and always flowing.

Data comparison needs to be done only for those data that are updated at least 3 minutes ago in DynamoDB. (to ensure the same had been arrived and updated in RDS)

asked a year ago130 views

2 Answers
1

I don't think there is any efficient way of doing this. The most efficient thing you can do is to ensure your streaming logic from DynamoDB to RDS is robust and handles failures elegantly.

You can then periodically takes dumps of DynamoDB and RDS and compare them in an offline manner.

The other approach, would be to also stream your DynamoDB changes to an SQS queue, and process them changes with a delay of 3 minutes, and then read those events back from RDS to make sure it was committed.

Enter image description here

AWS
EXPERT

answered a year ago

0

You can leverage DynamoDB streams to get the MODIFY events.

DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours. Applications can access this log and view the data items as they appeared before and after they were modified, in near-real time.

You can then use a Lambda trigger or a custom stream consuming logic to compare the records from stream with your RDS rows.

AWS
EXPERT

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.