How do I troubleshoot AWS DMS endpoint connectivity failures?

Short description

You might receive two types of errors when you test the connection from the replication instance to the source or target endpoint: connectivity issues and native database issues. You must resolve these issues before your test can complete successfully.

Resolution

Connectivity issues

If the error occurred because of connectivity issue between replication instance and source or target, then you receive errors similar to the following:

• "Application-Status: 1020912, Application-Message: Failed to connect Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: HYT00 NativeError: 0 Message: [unixODBC][Microsoft][ODBC Driver 13 for SQL Server]Login timeout expired ODBC general error."
• "Application-Status: 1020912, Application-Message: Cannot connect to ODBC provider Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: 08001 NativeError: 101 Message: [unixODBC]timeout expired ODBC general error."
• "Application-Status: 1020912, Application-Message: Cannot connect to ODBC provider ODBC general error., Application-Detailed-Message: RetCode: SQL_ERROR SqlState: HY000 NativeError: 2005 Message: [unixODBC][MySQL][ODBC 5.3(w) Driver]Unknown MySQL server host 'mysql1.xxxxx.us-east-1.rds.amazonaws.com' (22) ODBC general error."

Native database error

If the error occurred because of a native database error, such as database permission or authentication error, then you receive errors similar to the following:

• "Application-Status: 1020912, Application-Message: Cannot connect to ODBC provider Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: 08001 NativeError: 101 Message: [unixODBC]FATAL: password authentication failed for user "dmsuser" ODBC general error."

Note: It's a best practice to test the connectivity from the AWS DMS replication instance to the endpoints after you create your AWS DMS source and target endpoints. Complete the test before you start the AWS DMS migration task. Otherwise, the task might fail because of connectivity issues with the endpoint.

Resolve connectivity issues for AWS hosted resources

Verify that you can establish connectivity between the source or target database and the replication instance. Depending on your use case and network infrastructure, connect your source or target database to a replication instance in a public subnet or private subnet. For more information, see Setting up a network for a replication instance.

Note: AWS DMS versions 3.4.7 and later require that you configure AWS DMS to use virtual private cloud (VPC) endpoints. Or, you must use public routes to all your source and target endpoints that interact with certain Amazon Web Services. If your AWS DMS endpoint tests are successful on earlier versions but fail on later versions, then see Configuring VPC endpoints as AWS DMS source and target endpoints.

Check your replication instance configuration

In your replication instance, confirm that your configuration includes the following:

• An Outbound Rule for the IP address with the port of the source or target database in the security group. By default, the Outbound Rule of a security group allows all traffic. Security groups are stateful, so you don't need to modify the Inbound Rule from the default.
• An Outbound Rule for the IP address with the port of the source or target database in the network ACL. By default, the Outbound Rule of a network access control list (ACL) allows all traffic.
• An Inbound Rule for the IP address with the ephemeral ports of the source or target database in the network ACL. By default, the Inbound Rule of a network ACL allows all traffic.

Check your source or target database configuration
In your source or target database, confirm that your configuration includes the following:

• An Inbound Rule for the IP address of the replication instance or the Classless Inter-Domain Routing (CIDR) of the subnet group of the replication instance. The Inbound Rule must include the port of the source or target database in the security group. Security groups are stateful, so you don't need to modify the Outbound Rule from the default.
  Note: To find the IP addresses and CIDRs, see the Determine the IP addresses and CIDR of a subnet group section.
• An Inbound Rule for the IP address of the replication instance or the CIDR of the subnet group of the replication instance. The Inbound Rule must include the port of the source or target database in the network ACL. Confirm that there is no explicit deny rule for the IP address and port allowed.
• An Outbound Rule for the IP address or the CIDR of the subnet group of the replication instance with ephemeral ports in the network ACL. By default, the Outbound Rule of a network ACL allows all traffic.
• It's a best practice to configure your network to allow the CIDR of the subnet group of the replication instance. The IP address of the replication instance changes during a failover or host replacement event.

Determine the IP addresses and CIDR of a subnet group

To set up inbound and outbound rules, determine the IP addresses and CIDR of the subnet group. You can either use the AWS DMS console or the AWS CLI.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

AWS DMS console

Complete the following steps:

1. Open the AWS DMS console.
2. In the navigation pane, choose Replication instances.
3. Choose the name of your replication instance.
4. Under Details, copy the Public IP address, Private IP address, and the Replication subnet group of your replication instance.
5. Under Replication subnet group, choose the link to access the subnet group page. Copy the name of each subnet in the subnet group.
6. To verify the CIDR of each subnet open the Amazon Virtual Private Cloud (Amazon VPC) console.
7. In the Subnets tab, search for the subnets noted in step 5. For each subnet, note the CIDR.

AWS CLI

To determine the CIDR of each subnet, run the describe-subnets command.

aws ec2 describe-subnets --filters Name=subnet-id,Values="$(aws dms describe-replication-instances --filters "Name=replication-instance-id,Values=replication-instance-name" --query "ReplicationInstances[*].ReplicationSubnetGroup.Subnets[*].SubnetIdentifier" --output text | sed -e 's/\t/,/g')" --query "Subnets[*].{SubnetId:SubnetId,CidrBlock:CidrBlock}" --output table

Note: Replace replication-instance-name with the name of your replication instance.

To determine the IP addresses of the replication instance, run the describe-replication-instances command.

aws dms describe-replication-instances --filters "Name=replication-instance-id,Values=replication-instance-name" --query "ReplicationInstances[*].{ReplicationInstancePublicIpAddresses:ReplicationInstancePublicIpAddresses,ReplicationInstancePrivateIpAddresses:ReplicationInstancePrivateIpAddresses}" --output table

Note: Replace replication-instance-name with the name of your replication instance.

Resolve connectivity issues (on-premises resources)

If your source or target database is hosted on-premises, then confirm the following:

• The database allows incoming connections from the AWS DMS replication instance. To confirm, check with your network administrator.
• A firewall doesn't block communication to the source or target database.
• The DNS configuration is set up correctly. If you require DNS resolution, then use the Amazon Route 53 Resolver. For information on how to use an on-premises name server to resolve endpoints with the Amazon Route 53 Resolver, see Using your own on-premises name server.
  -or-
  Create a new AWS DMS instance through the AWS CLI to use a customer DNS name server (--dns-name-servers) to resolve DNS issues. By default, AWS DMS instances use Amazon-provided DNS for resolutions. AWS DMS endpoints might fail when the source or target is configured to use custom DNS. For more information, see create-replication-instance.

Make sure that your Amazon Elastic Compute Cloud (Amazon EC2) instance has the same network configurations as the AWS DMS replication instance with the connectivity issues. To troubleshoot network connectivity, first run the following commands on the new Amazon EC2 instance:

telnet database_IP_address_or_DNS port_number

Note: Replace database_IP_address_or_DNS with the IP address or domain name of the database specified for the AWS DMS source or target endpoint. Replace port_number with the port number of the database specified for the AWS DMS source or target endpoint.

Then, run the following command:

nslookup domain_name

Note: Replace domain_name with the domain name of the database specified for the AWS DMS source or target endpoint.

Resolve native database errors

To resolve native database errors, confirm that the following endpoint configurations are set correctly:

• Username
• Password
• The ServerName is set to the DNS or IP of the on-premises database, or Amazon Relational Database Service (Amazon RDS) endpoint
• Port
• Database name
  Note: Don't specify a Database name for MySQL source or target.

If any of these fields are specified with AWS Secrets Manager, then see Using secrets to access AWS Database Migration Service endpoints.

For native database errors related to the source or target database, see the resolution from the specific database documentation. Use the error code and the error message that you receive in the AWS DMS console.

For more information, check the error, trace, alert, or other logs from the source or target database.

For database access errors, confirm the permissions required by AWS DMS for the specific source or target.

For more information on how to encrypt connections for source and target endpoints with SSL, see Using SSL with AWS Database Migration Service.

Related information

How can I troubleshoot Amazon S3 endpoint connection test failures when using AWS DMS?

How can I troubleshoot connectivity failures and errors for an AWS DMS task that uses Amazon Redshift as the target endpoint?

How do I troubleshoot connectivity failures between AWS DMS and a MongoDB source endpoint?

Migrating Microsoft SQL Server databases to the AWS Cloud