- Le plus récent
- Le plus de votes
- La plupart des commentaires
Though I havn't myself done the setup end to end . At solution high level the below approach should work. Aurora Database Activity Streams with AWS Lambda and S3 https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/DBActivityStreams.Overview.html
1/ Enable Database Activity Streams: This built-in Aurora feature captures database modifications (inserts, updates, deletes) in near real-time. It automatically creates a Kinesis data stream to push this activity data.
2/ AWS Lambda Function: Create a Lambda function triggered by the Kinesis data stream. This function will process the activity stream events and extract the relevant data changes.
3/ Data Transformation (Optional): The Lambda function can transform the extracted data if needed to match your desired format for historical storage in S3.
4/ Store Versions in S3: Use the AWS SDK within the Lambda function to write the extracted and potentially transformed data to S3 objects. Each object can represent a specific version of your data.
OR maybe , some thing similar AWS CDC (Change Data Capture) with Amazon EventBridge and S3 : https://aws.amazon.com/blogs/database/capturing-data-changes-in-amazon-aurora-using-aws-lambda/
Hope it helps.
Contenus pertinents
- demandé il y a un an
- demandé il y a un an
- AWS OFFICIELA mis à jour il y a un an
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a 2 ans
Thank you for your answer.
For Asynchronous Database Activity Streams, the error handling doesn't seem to be straightforward (https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/DBActivityStreams.Overview.html#DBActivityStreams.Overview.sync-mode). It's saying we would receive an RDS event; however, I'm unsure how we can use that to reprocess events. Additionally, failures could potentially disrupt the order of events (I forgot to mention this in my question, but the ordering of events is critical for my use case). Please let me know if you are aware of how we can handle this gracefully.
Regarding the asynchronous lambda trigger approach, I really don't want to deal with triggers because they create a lot of unmanageable configurations in the database.