Amazon RDS (MySql) export to s3 - Is there any way to provide partition keys to each for the tables so that the S3 data falls into these dynamic partitioned folders created for each table within DB

0

Amazon RDS (MySql) export to s3. Is there any way to provide partition keys to each for the tables within the database. i would like the data falling into s3 as parquet files and partitioned on a date column within the data as for ex: s3://<tablename>/<snapshot-date>/year=2022/month=07/day=22 ? is it possible to have a dynamic S3 prefix created for each of the daily snapshot exported?

2 Answers
0

A glue ETL job can do that for you .

  1. Read the DMS data
  2. Write the data back to a different S3 bucket based on your partition keys .
  3. Loop it for each table through a Glue Python job

On the high level its a 2 step process , minimal coding , use DynamicFrames which will also help you in clubbing multiple CDC files into larger chunks improving better reads on the consuming applications .

Please note

  1. Glue will incur costs
  2. Its not going to be real time as DMS , since additional read and writes are involved
  3. DMS will result in multiple versions of the record , For example , Inserts , Multiple Updates and Delete . You can handle that in Glue, but the program will have a few more steps then. This is based on your requirement though
answered 2 years ago
0

was this solution implementeD?

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions