- Newest
- Most votes
- Most comments
It appears that you're encountering an issue with your AWS Database Migration Service (DMS) task where the Change Data Capture (CDC) files in your Amazon S3 target are missing column names, despite having set AddColumnName to true. This is an unexpected behavior, as the AddColumnName setting should typically include column names in both full load and CDC files.
To address this issue, there are a few things you can check and try:
-
Verify your endpoint settings: Double-check that the AddColumnName parameter is indeed set to true in your S3 target endpoint settings. Sometimes, settings may not apply correctly if there are conflicts or if they weren't saved properly.
-
Check compatibility with other settings: The AddColumnName parameter cannot be used together with the CdcPath parameter. If you're using CdcPath to specify a custom location for CDC files, this might be causing the conflict. You may need to choose between using AddColumnName or CdcPath, depending on your priorities.
-
Review your task settings: Ensure that your task is configured correctly for CDC. Make sure that you have enabled ongoing replication (CDC) in your task settings and that the task has successfully transitioned from full load to CDC phase.
-
Check source database configuration: Ensure that your source database is properly configured for CDC. For example, if you're using a MySQL-compatible database, make sure binary logging is enabled and properly configured.
-
Examine CDC file format: CDC files in S3 typically follow a specific format. They should include the operation type (INSERT, UPDATE, DELETE), table name, schema name, and then the data columns. If your CDC files are missing this structure entirely, there might be a more fundamental issue with how CDC is being captured or written.
-
Consider using date-based folder partitioning: Setting up date-based folder partitioning for your target can sometimes help with organizing and properly formatting CDC data.
-
Check DMS version: Ensure you're using a recent version of AWS DMS, as older versions might have bugs or limitations related to CDC file formatting.
If none of these suggestions resolve the issue, you may want to contact AWS support for further assistance, as this behavior seems unusual given your described configuration.
Sources
Using Amazon S3 as a target for AWS Database Migration Service - AWS Database Migration Service
Using Amazon S3 as a source for AWS DMS - AWS Database Migration Service
Amazon S3 target data validation - AWS Database Migration Service
Here is the issue, if I remove CDCPath, I get this message
Test Endpoint failed: Application-Status: 1020912, Application-Message: cdcpath can't be empty
But if I add it, I cannot add AddColumnName ? These two parameters are important if needing to replicate data and then consume the data from external sources. Am I the only one that has ever attempted this?
As per AWS documentation mentioned below, you can't use addColumnName together with PreserveTransactions or CdcPath.
You will need to set both PreserveTransactions and CdcPath to false if you want to use addColumnName.
[+] limitations to using Amazon S3 as a target - https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html#CHAP_Target.S3.Limitations
Relevant content
- asked 8 months ago
- asked 2 years ago
- Accepted Answerasked 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago