Why is my AWS DMS replication DB instance in the storage-full status?

Short description

AWS DMS replication DB instances have storage, and you can specify the allocated storage when you create a replication DB instance. By default, the dms.t2.* and dms.r4.* replication DB instance types have volume sizes of 50 GiB. The dms.c4.* replication DB instance types have volume sizes of 100 GiB. Monitor the Amazon CloudWatch metric for FreeStorageSpace to monitor the free storage space of the replication DB instance.

Replication DB instance storage is used mostly for task logs and cached changes that are collected for ongoing replication.

Migrations that have a full load only

Task logs can cause storage-full issues if there are lots of task logs, for example, if you turned on detailed debug logging for a task. By default, the AWS DMS task logging level is minimal, but sometimes you need more detailed logging to determine the cause of a migration error or failure. But detailed debug logging can consume replication storage because AWS DMS writes each and every event to the log files. Also, too many task logs can cause your replication DB instance to have a storage-full status.

Migrations that have ongoing replication

Cached events can cause storage issues during a full load phase, especially if you configure your migration to use the StopTaskCachedChangesNotApplied task setting. For more information, see Working with an AWS DMS replication instance. If the storage size is too small to store the cached changes, then your replication DB instance has a storage-full status.

Swapped events during the change data capture (CDC) phase can cause storage issues when the changes captured from the source can't be applied to the target. AWS DMS first stores the cached events in memory, and then swaps them to storage when the values defined by the MemoryKeepTime or MemoryLimitTotal parameters are exceeded (60 seconds and 1 GiB by default, respectively). If there isn't enough storage for the swapped data, the replication DB instance has a storage-full status. For more information, see Change processing tuning settings.

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent version of the AWS CLI.

Delete the task logs

If the replication DB instance is in an available status, you can delete the task logs:

1. Open the AWS DMS console.
2. Choose Replication instances from the navigation pane.
3. Choose the name of the replication DB instance.
4. From the Log management section, you see all the tasks and the Log size of each task.
5. Select the tasks for which you want to delete logs, and then choose Delete.

For more information, see Viewing and managing AWS DMS task logs.

Use CLI to modify the task setting

You can also delete the DMS task logs using the AWS Command Line Interface (AWS CLI) by modifying the task setting. First, create a tasksetting.json JSON file using the following command:

{
  "Logging": {
    "DeleteTaskLogs":true
  }
}

After you have created the JSON file, run the command below to modify the task setting.

Note: The task must have a status of Stopped or Failed to be modified.

aws dms modify-replication-task --replication-task-arn  --replication-task-settings file://tasksetting.json

The command above modifies the task setting by changing the parameter "DeleteTaskLogs": true. After you have added this parameter and value to the task settings, all of the logs present for the task are deleted from the replication DB instance. After the logs are deleted, the task setting removes the "DeleteTaskLogs": true parameter. After running this command, confirm that the logs are deleted from the replication DB instance and that the task setting removed the parameter "DeleteTaskLogs": true.

To delete the logs periodically, you must modify the task setting with the "DeleteTaskLogs": true parameter, every time. You can schedule this command to run periodically using cronjob or Lambda scheduler, as per your implementation.

For more information, see modify-replication-task.

Increase the storage size of replication DB instance

If the replication DB instance is in a storage-full status, you can't delete logs. You must increase the allocated storage size, or delete the replication task. To increase the storage size of a replication DB instance:

1. Open the AWS DMS console.
2. Choose Replication instances from the navigation pane.
3. Choose the name of the replication DB instance that you want to modify the volume size for.
4. Choose Modify.
5. For the Allocated storage (GB) field, enter a new value.
6. Select Apply changes immediately, and choose Modify.

Important: The allocated storage size can't be decreased after it has been increased.

Related information

Best practices for AWS Database Migration Service