Amazon RDS (MySql) export to s3 - Is there any way to provide partition keys to each for the tables so that the S3 data falls into these dynamic partitioned folders created for each table within DB

0

Amazon RDS (MySql) export to s3. Is there any way to provide partition keys to each for the tables within the database. i would like the data falling into s3 as parquet files and partitioned on a date column within the data as for ex: s3://<tablename>/<snapshot-date>/year=2022/month=07/day=22 ? is it possible to have a dynamic S3 prefix created for each of the daily snapshot exported?

2 Antworten
0

A glue ETL job can do that for you .

  1. Read the DMS data
  2. Write the data back to a different S3 bucket based on your partition keys .
  3. Loop it for each table through a Glue Python job

On the high level its a 2 step process , minimal coding , use DynamicFrames which will also help you in clubbing multiple CDC files into larger chunks improving better reads on the consuming applications .

Please note

  1. Glue will incur costs
  2. Its not going to be real time as DMS , since additional read and writes are involved
  3. DMS will result in multiple versions of the record , For example , Inserts , Multiple Updates and Delete . You can handle that in Glue, but the program will have a few more steps then. This is based on your requirement though
beantwortet vor 2 Jahren
0

was this solution implementeD?

beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen