One DMS Endpoint is Failing to Find Existing Secret

Question

There exists many DMS Endpoints that are able to access RDS instances using a Secrets Manager Secret.

There is one endpoint that is configured like all the rest, however, when the test-connection command is issued the result is an error.

> Test Endpoint failed: Application-Status: 1020912, Application-Message: Cannot connect to SQL Server Unable to find Secrets Manager secret, Application-Detailed-Message: Failed to retrieve secret. Unable to find AWS Secrets Manager secret Arn 'arn:aws:secretsmanager:us-east-1:XXXXXXXXX:secret:XXX-tsm-rpt-blu-use1-rds-master-secret-Jkf6ugXj-vFN1l7'

Another task using the same endpoint failed with a slightly different error:

> Test Endpoint failed: Application-Status: 1020912, Application-Message: Cannot connect to SQL Server Unable to find Secrets Manager secret, Application-Detailed-Message: Failed to retrieve secret. Unable to find AWS Secrets Manager secret Arn 'arn:aws:secretsmanager:us-east-1:XXXX:secret:XXX-tsm-rpt-blu-use1-rds-master-secret-Jkf6ugXj-vFN1l7' The secrets_manager get secret value failed: curlCode: 7, Couldn't connect to server Too many retries: curlCode: 7, Couldn't connect to server

The  list-secrets command returns a secret with the arn used in the failing Endpoint.
**dms list-secrets**

```

...
 {
    "ARN": "arn:aws:secretsmanager:us-east-1:XXXX:secret:XXX-tsm-rpt-blu-use1-rds-master-secret-Jkf6ugXj-vFN1l7",
    "Name": "XXX-tsm-rpt-blu-use1-rds-master-secret-Jkf6ugXj",
     ...
 }
...
```

The get-secret-value command returns the value of the secret.
**dms get-secret-value --secret-id XXX-tsm-rpt-blu-use1-rds-master-secret-Jkf6ugXj**

```
 {
    "ARN": "arn:aws:secretsmanager:us-east-1:986204609104:secret:crem-tsm-rpt-blu-use1-rds-master-secret-Jkf6ugXj-vFN1l7",
    "Name": "crem-tsm-rpt-blu-use1-rds-master-secret-Jkf6ugXj",
    "VersionId": "52C346EC-13B1-46A8-946E-665C3B1559CF",
    ...
 }

```

I don't think this is a network configuration issue because I would expect to see a timeout error, plus other endpoints are able to retrieve the secret.

Any ideas on what the issue could be?

Here is the policy being used for this:

```
{
    "Statement": [
        {
            "Action": "secretsmanager:GetSecretValue",
            "Effect": "Allow",
            "Resource": "arn:aws:secretsmanager:us-east-1:xxxx:secret:xxx-tsm-rpt-blu-use1-rds-master-secret-Jkf6ugXj-vFN1l7"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kms:us-east-1:xxxx:key/xxxx"
        }
    ],
    "Version": "2012-10-17"
}
```
There is an iam role with the above policy attachment with an assume role:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "dms.us-east-1.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
```

The iam role above is attached to the DMS Endpoint.

There also exists a well configured VPC endpoint for com.amazonaws.us-east-1.secretsmanager.

Other endpoints, configured the same way, are able to access the RDS instances without error.

Accepted Answer

Hi,

The first thing is to check the DMS IAM policy,  The secret:getSecret must be specified. If you work in  a private network, Do not forget to create a Secret manager endpoint. This should fix your issue.

One DMS Endpoint is Failing to Find Existing Secret

Relevant content