DMS r5.2xlarge stuck at 0.06% CPU with 974k CDCChangesDiskTarget and INTERNAL (54) Sorter error

Question

Problem Summary: My DMS task is experiencing a persistent hang during the CDC phase. Despite using an r5.2xlarge replication instance (64GB RAM), the CPU Utilization is flatlining at 0.06% and the task is not processing any changes.

Environment:
Replication Instance: r5.2xlarge (Engine 3.4.7 or 3.5.x)
Source: [MySQL 8.0]
Target: Aurora [MySQL 8.0]

CDC Latency: 14.5 hours and growing.

Evidence of Hang from Logs: The task logs consistently repeat the following entries every 30 seconds: [SORTER ]I: Task is running {operation:INTERNAL (54)} (sorter.c:795) [TASK_MANAGER ]D: There are 1 swap files of total size 667 Mb. Left to process 1 of size 667 Mb (replicationtask_cmd.c:1962)

Metrics observed in CloudWatch:
CDCChangesDiskTarget: Constant at ~974,000.
FreeableMemory: ~60 GB (Instance is not memory-constrained).
CPUUtilization: 0.06% (Instance is idle).
DiskReadBytes/WriteBytes: 0 (No I/O activity despite the swap file).

Steps Taken:
Increased MemoryLimitTotal to 51,200 MB and BatchApplyMemoryLimit to 30 GB (and later reduced to 2 GB).
Rebooted the target RDS instance.
Restarted and Resumed the DMS task multiple times.
Verified Primary Keys exist on all tables.

Question: The Sorter component appears deadlocked with the Target Apply worker (INTERNAL 54 error). Since there is 60GB of free memory, why is the Sorter not pulling the 667MB swap file into memory? Is this a known bug in the Sorter's handling of disk-spilled transactions, and is there a way to force a flush without losing the current CDC position?

re:Post Agent · Answer

This issue appears to be related to the AWS DMS task caching transactions on the replication instance, which is indicated by the high CDCChangesDiskTarget value (~974,000) and the SORTER message in your logs.

When a target endpoint can't keep up with the volume of changes that AWS DMS writes to it, the task caches the changes on the replication instance. If the cache grows larger than an internal threshold, the task stops reading further changes from the source. DMS does this to prevent the replication instance from running out of storage or getting stuck while reading a large volume of pending events.

The INTERNAL (54) error and the message about swap files indicate that DMS has paused reading from the source because the target can't keep up with processing the changes. Despite having sufficient memory available, the system is likely experiencing a bottleneck at the target endpoint.

To troubleshoot this issue:

1. Check if there are any indexing issues on the target database that might be slowing down writes.
2. Look for potential database locking on the target that could be blocking DMS write transactions.
3. Consider moving tables with numerous transactions or long-running DML operations to a separate task.
4. You might need to further increase the `MemoryLimitTotal` and `MemoryKeepTime` settings.
5. Evaluate if batch apply can be used more effectively for your transactions.

The low CPU utilization (0.06%) suggests that the task is indeed waiting on something else, likely the target database's ability to process the changes. This is consistent with a target latency issue where DMS is reading data from the source smoothly but encountering a bottleneck when writing to the target.

To resolve this, you may need to investigate the target database's performance, optimize indexes, or consider scaling up the target database resources to handle the write load more efficiently.  
**Sources**  
[Troubleshooting target latency issues - AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Troubleshooting_Latency_Target.html)  
[Troubleshooting latency issues - AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Troubleshooting_Latency_Troubleshooting.html)

DMS r5.2xlarge stuck at 0.06% CPU with 974k CDCChangesDiskTarget and INTERNAL (54) Sorter error

Add your answer

Relevant content