AWS DMS Aurora MYSQL Serverless V2 As A source Failing to Replicate New Columns

Question

I am using AWS DMS to replicate data from an Aurora MYSQL Serverless V2 instance to a S3 destination.  The task is a full load with CDC.  It is working fine and replicating all inserts, updates and deletes.  However when I add a new column to a table we get the following error and the data stops replicating to S3.

Mismatch number of columns in table Test.MyTable: CDC event has 7 columns while Replicate knows 6. Only the first 6 columns will be replicated. This may imply that DDL changes were done and reverted when the task did not run.

Answer

I would recommend you to look into the below workaround to resolve the issue.

a) In some cases, the source table will not be fetched by DMS, such as through tools such as Percona pt-online-schema-change A similar alarm appears when the field is modified. Although you may not use such a tool, it is recommended that you modify the source endpoint and add Extra connection attributes(ECA) “cleanSrcMetadataOnMismatch= true” [1], which clears and rebuilds the table metadata cache when the source table structure is not the same as the replicated instance cache table structure to avoid similar problems.

b) Create a new task with same settings and mapping rules and check if you're still facing the issue. Also, enable detailed debugging logs on source_capture, source_unload, target_load, target_apply and task_manager components.[2]

References :
=========
[1] Extra connection attributes when using MySQL as a source for AWS DMS - https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html#CHAP_Source.MySQL.ConnectionAttrib

[2] Logging task settings  - https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.Logging.html

Please do not hesitate to contact the AWS Support team if the above workaround doesn't help. We would be happy to help.

[+] https://console.aws.amazon.com/support

AWS DMS Aurora MYSQL Serverless V2 As A source Failing to Replicate New Columns

References :

Relevant content