Hello AWS Community,
I am currently storing event logs in an RDS Postgres database and am looking for an efficient way to manage the growing size of our tables. Here's what I am aiming to achieve:
-
Data Migration and Deletion: I want to migrate data that's more than one day old from the Postgres database to S3, and store it in Parquet format. After migration, the data should be deleted from the Postgres table to prevent it from becoming too large.
-
Use of AWS Services: While it seems feasible to use AWS Glue for a similar task, my understanding is that it requires fetching the entire table and then applying filters to replicate the data. I'm curious if there's a way to optimize this process.
-
Looking for Managed Services: Ideally, I would like to use a managed service that can perform this operation efficiently, cost-effectively, and with minimal setup. Writing a custom batch script is an option, but I am wondering if there's an existing AWS service specifically designed for such a use case.
Could anyone advise if there is an AWS service that fits these requirements or share any best practices for handling such scenarios?
Thank you in advance for your assistance!
Best regards,
Alan.
