Skip to content

DMS Task does not put full-load file inside partition folders

0

I am using DMS to migrate data from PostgresSQL to S3 in parquet format and with date-based folder partitioning, but the full-load file is not generated inside a folder but in the root, while the CDC files are generated inside folders defined as partitions. I did not find anything in the documentation saying that this is expected behavior, hence my question, is this expected behavior or am I doing something wrong?

Documentation link https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html#CHAP_Target.S3.DatePartitioning

asked 2 years ago718 views

1 Answer
0

This behavior is actually expected when using AWS Database Migration Service (DMS) to migrate data from PostgreSQL to Amazon S3 with date-based folder partitioning. The full-load files and CDC (Change Data Capture) files are handled differently in terms of partitioning.

For full-load data, DMS typically writes the files directly to the root of the specified S3 bucket or folder path. These files are not partitioned based on date because they represent the initial snapshot of your data at a single point in time.

On the other hand, CDC files, which capture ongoing changes after the initial full load, are written to the date-based partitioned folders. This is because CDC data is associated with specific timestamps of when the changes occurred.

The reason for this difference is that date-based folder partitioning in DMS is primarily designed to organize and manage the ongoing replication data (CDC) more effectively. It helps in better management of S3 objects, limits the size of each S3 folder, and optimizes subsequent data lake queries or operations.

If you want to have your full-load data partitioned as well, you might need to consider additional steps after the DMS task completes, such as using AWS Glue or other ETL processes to reorganize the full-load data into the desired partition structure.

It's worth noting that while this behavior might not be explicitly stated in the documentation you referenced, it is consistent with how DMS typically handles full-load and CDC data when writing to S3 targets with partitioning enabled.

Sources
Step-By-Step Migration - Database Migration Guide
Amazon S3 target data validation - AWS Database Migration Service

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.