How do I automate the AWS DMS error log deletion process to avoid storage-full on my replication instance?

5 minute read

I want to automate the AWS Database Migration Service (AWS DMS) error log deletion process to delete more often. How can I do this?

Short description

If your AWS DMS error logs take up a lot of space in your DMS replication instance, the replication instance might go into the storage-full state. To fix this, delete the DMS task error logs for your DMS tasks.

You can delete the error log for your AWS DMS task using one of the following methods:

Using the AWS DMS console
Using the AWS Command Line Interface (AWS CLI)
Using API (implemented with Python in this example)

Note: There are a number of other reasons why storage-full can occur. For more information on resolving storage-full issues with AWS DMS, see Why is my AWS DMS replication DB instance in the storage-full status?

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

Delete error log using the AWS DMS Console

To delete the error log of your DMS task manually, see How do I enable, access, and delete Amazon CloudWatch logs for AWS DMS?

Delete error logs using the AWS CLI

You can delete your DMS task logs by modifying your task settings using the following AWS CLI command:

aws dms modify-replication-task --replication-task-arn <DMS task ARN> --replication-task-settings '{"Logging": { "DeleteTaskLogs": true}}'

This command modifies the task by changing the "DeleteTaskLogs": true parameter. After you add the task setting with this parameter and value, all logs present for this task are deleted from the replication instance. After the logs are deleted, the task settings remove the "DeleteTaskLogs": true parameter. After running this command, confirm that the logs are deleted from the replication instance.

To delete the logs periodically, modify the task setting with the "DeleteTaskLogs": true parameter, every time. You can schedule this AWS CLI command to run periodically using cronjob. For more information about modifying task setting, see modify-replication-task in the AWS CLI Reference.

Automate task log deletion process using API (Lambda)

AWS DMS task logs are deleted every seven days for all replication instances. To delete the logs more often, you can create a script and automate the process to run every day, or at the frequency you want. This example uses simple Python code to automate the task error log deletion for the provided task ARN using AWS Lambda.

Set up the Lambda function

To set up the Lambda function, perform the following steps:

1. Open the Lambda console, and then choose the AWS Region containing your DMS resources.

2. From the Functions panel, choose Create function.

3. Enter a Function name.

4. For Runtime, choose python 3.8.

5. For Change default execution role, choose Create a new role with basic Lambda permissions. Note the AWS Identity and Access Management (IAM) role name that Lambda creates.

6. Choose Create function.

7. Open the AWS IAM console, and then open the IAM role that the Lambda function created.

8. Create an IAM policy using the following JSON. Make sure to replace the task ARN for Resource:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dms:ModifyReplicationTask"
            ],
            "Resource": "arn:aws:dms:us-east-1:1234567890:task:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
        }
    ]
}

9. Attach the newly created policy to the role created by the Lambda function.

10. Open the Lambda console, and then choose the Lambda function you created.

11. For Function code, enter the following code, and then click Deploy:

import boto3
import json
client = boto3.client('dms')
def lambda_handler(event, context):
    # specific task ARN
    taskarn = 'arn:aws:dms:us-east-1:1234567890:task:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
    # modify the DMS task
    response = client.modify_replication_task(
        ReplicationTaskArn=taskarn,
        ReplicationTaskSettings='{"Logging": { "DeleteTaskLogs": true}}'
    )

12. Choose Test, and then enter Event name with the default template.

13. Choose Create.

14. Choose Test again, and the code runs without errors.

After performing these steps, the DMS task status changes to Modifying. At the same time, you can see that the DMS task error logs under the replication instance are deleted.

Automate using Lambda Scheduler

To automate the deletion of DMS task error logs using Lambda Scheduler, perform the following steps:

1. Open the Lambda console, and then choose the Lambda function you created.

2. For Designer, choose Add trigger.

3. From the dropdown list, choose EventBridge (Cloudwatch Events).

4. For Rule, choose Create a new rule.

5. Enter Rule name and Rule description.

6. To run the code every seven days, for schedule expression, enter rate (7 days). For more flexible expressions, see Schedule expressions using rate or cron.

7. Choose Add.

After completing these steps, the automation is set up to delete the specified DMS logs once every seven days.

Note: You can change and optimize this code as per your business requirement. You can also schedule this script to run periodically using cronjob or Lambda scheduler.