How do I manage AWS DMS engine version upgrades, availability, and compatibility across Regions?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Check for available AWS DMS versions

AWS delivers new releases to different Regions at different times. To check the availability of the AWS DMS versions in your Region, it's a best practice to use the AWS CLI and the AWS DMS support lifecycle policy.

To check available versions in your Region, run the following describe-orderable-replication-instances AWS CLI command:

aws dms describe-orderable-replication-instances

Check the end of life dates for AWS DMS versions

After AWS DMS releases a new version, check the AWS DMS release notes for the end of life (EOL) dates of your current DMS version. Or, check the AWS DMS support lifecycle policy to prepare for the upgrade.

It's a best practice to regularly review AWS What's New, AWS News Blog, and AWS Database Blog for details on new releases.

If you run an AWS DMS version that's no longer supported, then your AWS Health Dashboard sends you a reminder to upgrade. AWS might force an upgrade on the instances that you don't upgrade after the EOL date.

Prepare for the upgrade

Prerequisites:

• Review the AWS DMS release notes for the version that you want to upgrade to and make sure that the source and target are compatible.
• Turn on AWS DMS data validation for supported source and target endpoints to make sure that your data migrates accurately.
• It's a best practice to perform the version upgrade on a TEST/UAT instance first, and then validate the performance.
• Before you upgrade the version, it's a best practice to run the stop-replication-task AWS CLI to manually stop the AWS DMS tasks. Then, resume or restart the tasks after you updated the version.
• To avoid the upgrade during working hours, schedule the version upgrade during a maintenance window.
• For large tables, create more than one task to reduce the effect of a single task issue.
• If you use the current default engine version when you create a replication instance, then use the automatic version upgrade option. AWS DMS can then perform the upgrade during the maintenance window.

To check the database (DB) instance class support for the AWS DMS version, run the following describe-orderable-replication-instances AWS CLI command:

aws dms describe-orderable-replication-instances \
--query "OrderableReplicationInstances[*].[ReplicationInstanceClass,EngineVersion]"

Perform the upgrade

To upgrade the AWS DMS Replication instance version, see Working with replication engine versions.

To monitor the version upgrade status, run the following describe-replication-instances AWS CLI command:

aws dms describe-replication-instances --query "ReplicationInstances[*].[ReplicationInstanceIdentifier,ReplicationInstanceStatus]"

To reduce downtime and maintain a stable replication environment for instances with a large number of tasks, it's a best practice to use a side-by-side upgrade.

Review your task performance

If you manually stopped the AWS DMS task and the instance is in AVAILABLE state, then you must restart or resume the AWS DMS tasks. Use Amazon CloudWatch metrics to monitor the task level and replication instance metrics for failures, slow performance, and latency issues. To compare the instance's performance, review the CloudWatch metrics.

It's a best practice to turn on monitoring for AWS DMS tasks, and notify stakeholders of upgrade errors. For more information about how to set up monitoring at the task level, see Automating database migration monitoring with AWS DMS.

Troubleshoot high latency

If your AWS DMS tasks have higher latency or performance issues after a version upgrade, then take the following actions:

• To check the AWS DMS task logs for ERRORS/WARNINGS that cause a task to fail, search for "]E:" and "]W:" in the task logs of the CloudWatch log group for a specific task.
• Check the replication instance level, source, and target metrics for resource throttling. For more information, see AWS Database Migration Service metrics.
• Confirm that the new replication engine version is compatible with the previous source and target engine versions.
• Make sure that the log retention period is larger than the default value for the migration source.
• Compare the CDCLatencySource, CDCLatencyTarget, and CDCIncomingChanges replication task metrics with the task metrics from before you performed the version upgrade.

Also, see the following documentation:

• How do I troubleshoot high source latency on an AWS DMS task?
• How can I troubleshoot high target latency on an AWS DMS task?
• Troubleshooting latency issues in AWS Database Migration Service

If you still have high latency without a workload change, then contact AWS Support.

Troubleshoot endpoint connection failures

If you hardcode the IP address of source or target database engines in the endpoint settings, then you might get a connection error message.

If your endpoint connection fails after a version upgrade, then take the following actions:

• Check for failed endpoints connection test error messages on the AWS DMS console.
• Check the network connectivity between the source, target, and AWS DMS instance. Make sure that you attached the correct network access control lists (network ACLs) and security groups, and set the correct inbound and outbound address and ports.
• Validate database permissions and authentication credentials for AWS DMS users, and make sure that they can connect and migrate data.
• Review the SSL/TLS configuration for your endpoint.

For more information, see How do I troubleshoot AWS DMS endpoint connectivity failures?

Troubleshoot instance class error

If you upgrade your instance class in a maintenance window and you don't have enough capacity, then you get the following error message:

"ERROR : "Cannot modify the DB instance because there is not sufficient capacity in the required availability zone(s)""

This error occurs when there's a capacity issue in the Availability Zone where your AWS DMS replication instance is located. To resolve this issue, run your instance in a different Availability Zone.

It's a best practice to create a Multi-AZ deployment for high availability and failover support. If there's an Availability Zone failure, then a replication instance from another Availability Zone continues the data replication.

Troubleshoot out of memory errors

If your AWS DMS tasks fail after a version upgrade because of out of memory (OOM) errors, then take the following actions:

• If the task is in RUNNING state, then use the MemoryUsage and MemoryUsageBytes CloudWatch metrics to monitor memory use for the AWS DMS task.
• Check the AWS DMS task settings for parameter changes that you made at the memory level such as, MemoryKeepTime, MemoryLimitTotal, and validation settings.
• Check the replication instance metrics, such as, FreeableMemory, FreeMemory, SwapUsage and AvailableMemory.
• Compare the current task runtime data with the previous task runtime to determine whether memory use is higher.

Note: If no tasks are in RUNNING state, then FreeableMemory drops gradually over time and reclaims. To check whether there are memory leaks, contact AWS Support.

Troubleshoot table errors

Take the following actions:

• Check the control table for dmslogs.awsdms_apply_exceptions on the target database.
• For business-critical tables, review the table schema and data, and then configure error handling task settings.