The storage space on my AWS Database Migration Service (AWS DMS) DB instance is being consumed by swap files that have been generated on my instance. What are SWAP files, and how can I reduce swap file generation?
In AWS DMS, the SORTER component caches ongoing changes that come from the source. The component does this to manage actions such as transaction merging, filtering, and data transformations before applying them to the target.
If the source processing is over-performing the target apply, the target can't cope with incoming changes. Or, if long transactions exceed the maximum memory allocation (MemoryLimitTotal) and time (MemoryKeepTime), then the transactions in memory are spilled over to the replication instance disk, known as swap or sorter files.
Swap files like this are different from the OS swap files that are created when server memory isn't sufficient for processing. So, this data isn't included in the SwapUsage metric in Amazon CloudWatch.
You might see a message similar to this in your task logs:
"[SORTER ]I: Reading from source is paused. Total storage used by swap files exceeded the limit 1048576000 bytes (sorter_transaction.c:110) 00000849: 2021-04-28T07:19:20:987696 [SORTER ]I: Reading from source is resumed (sorter_transaction.c:116)]"
This is an information message, and not an error. You see this message when the swap files that are used by the AWS DMS SORTER component reaches a size limit of 1GB (hardcoded limit). AWS DMS versions 3.4.2 and later include a feature that prevents storage full scenarios by limiting the maximum amount of swap files you can have on a replication instance's storage volumes. So, you see this message when the swap files that are used by the DMS SORTER component reaches the 1 GB limit.
When this 1 GB limit is reached, the SORTER component pauses SOURCE CAPTURE. AWS DMS then applies changes from swap files to the target database. After the swapped changes from disk are applied or when the swap file drops below 1 GB, AWS DMS resumes source capture.
Note: When this happens, the task continues to run. Only SOURCE CAPTURE is paused.
Reduce swap file generation
Use these best practices to help reduce or avoid swap file generation on your AWS DMS instance:
- There are no primary keys or indexes in the target
- There are resource bottlenecks in the target
- There are resource bottlenecks in the replication instance
- There is a network issue between the replication instance and the target
- Use batch apply mode and ParallelApplyThreads to improve target apply, based on your engine type.
- If you are experiencing long running transactions at the source, then use frequent commits at the source database to help mitigate the issue.
- If memory is available on the replication instance, then you can increase MemoryLimitTotal and MemoryKeepTime so that changes remain in memory for longer before they are swapped to the disk. Make sure that you increase both of these parameters, because after a limit is reached, AWS DMS swaps the changes to the disk.
Change processing tuning settings