What are swap files and why do the files use space on my AWS DMS instance?

Resolution

AWS Database Migration Service (AWS DMS) generates swap files that use storage space as memory on a database instance. If the SOURCE_CAPTURE process produces faster than the TARGET_APPLY component, then the SORTER component caches the backlogged events to a disk or swap file. Also, the transactions in memory move to swap files on the replication instance disk if long transactions exceed the maximum memory allocation (MemoryLimitTotal) and time (MemoryKeepTime).

Note: AWS DMS swap files are different from operating system (OS) swap files, and Amazon CloudWatch doesn't include the swap files in the SwapUsage metric.

AWS DMS versions 3.4.2 and later include a feature that sets a 1 GB maximum storage for swap files on a replication instance's storage volumes. If the SORTER component swap files reach the maximum storage, then you might see the following error message in your task logs:

"[SORTER ]I: Reading from source is paused. Total storage used by swap files exceeded the limit 1048576000 bytes (sorter_transaction.c:110) 00000849: 2021-04-28T07:19:20:987696 [SORTER ]I: Reading from source is resumed (sorter_transaction.c:116)]"

Also, the SORTER component pauses SOURCE_CAPTURE. AWS DMS then applies changes from swap files to the target database. After AWS DMS applies the swapped changes or the swap file is less than 1 GB, AWS DMS resumes SOURCE_CAPTURE.

Note: While AWS DMS applies the changes, the task continues to run. The SORTER component pauses only SOURCE _CAPTURE.

Reduce swap file generation

To reduce or avoid swap file generation on your AWS DMS instance, check for causes of high target latency on your AWS DMS task.

To improve the TARGET_APPLY component's performance, set BatchApplyEnabled to True and use ParallelApplyThreads based on your engine type. For more information, see Target metadata task settings.

If you experience long-running transactions at the source, then use frequent commits at the source database. For more information, see How do I troubleshoot high source latency on an AWS DMS task?

When the replication instance has available memory, increase MemoryLimitTotal and MemoryKeepTime so that changes stay in memory longer before AWS DMS moves them to the disk. Make sure that you increase both parameters so that the SORTER component doesn't reach the quota.

Related information

Change processing tuning settings