How to specify s3 path for DMS table mapping (in json)

Question

I'm attempting to set up a DMS (Database Migration Service) task to migrate a single table from an Aurora MySQL database to Amazon S3. This migration involves multiple data transformations on that table. However, I've encountered an issue when trying to deploy this task via CloudFormation.

The error message indicates that my JSON file for table mappings exceeds the 4096 KB limit. While I understand there's a 2 MB maximum size for table mappings, I can't find a way to reference an S3 path for the TableMappings parameter when the file size is between 4096 KB and 2 MB.

Is there a method to specify an S3 location for the TableMappings parameter in this scenario? I'd appreciate any guidance on how to overcome this limitation and successfully deploy my DMS task with these larger table mappings.

Achyut_K · Answer

Kindly follow the answer given by Didier_Durand.

jiyukim · Answer

Hi,

As you mentioned in your question, AWS DMS can only process table mapping JSON files up to 2 MB in size. This is a hard limit. **And your JSON files, which is about 4MB, exceeds this hard limit.**

When a mapping rule file exceeds the 2 MB limit, we recommend that **you split the tables across multiple tasks to reduce the size of the mapping rule file so that it stays below this limit.**

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.html

Best,

Jiyu

Didier Durand · Answer

Hi,

This recent blog post is a solution to your problem: https://aws.amazon.com/blogs/database/aws-dms-best-practices-for-moving-large-tables-with-table-parallelism-settings/

Best,

Didier

How to specify s3 path for DMS table mapping (in json)

Add your answer

Relevant content