DMS Rejecting a Valid Secret Key Password

We have been running with DMS obtaining access to Target Endpoints (RDS SQL Server) via Secrets Manager in all environments for over a year without issue. We have a character exclusion mechanism on the secret rotator used to prevent a password with invalid chars from being generated. This runs in 5 environments without issue.

Three days ago, in a dev account environment, all endpoints targeting an RDS Instance has started failing. They fail on task reload/resume and test connection with the same error message:

2024-05-02T14:13:12.000Z	2024-05-02T14:13:12 [METADATA_MANAGE ]I: Retrieving AWS Secrets Manager secret Arn 'arn:aws:secretsmanager:us-east-1:XXXX::<secret_id>' content (secrets_manager_utils.c:413)

2024-05-02T14:13:13.000Z	2024-05-02T14:13:13 [METADATA_MANAGE ]E: In secret Arn arn:aws:secretsmanager:us-east-1:acct::<secret_id>, secret *key password contains one or more unsupported characters* [1027005] (secrets_manager_utils.c:445)

2024-05-02T14:13:13.000Z	2024-05-02T14:13:13 [METADATA_MANAGE ]E: Failed to retrieve secret. [1027005] (sqlserver_endpoint_imp.c:479)

NOTE: I can use the username and password obtained from get-secret-value using the key, above in the log, and successfully log into the RDS instance.

**NOTE: In the past, when an invalid char was generated during rotation, the log message would be similar to "could not connect username <blank> could not be authenticated...". After that, a manual secret rotation would rotate until a password matched a value that RDS SQL Server would accept. DMS never complained in the past about password chars (This is the first time in 3 years that I have seen this error). **

NOTE: The rotation lambda now excludes these chars from password generation

rotation_exclude_characters = " ",;?*!@=\/."

Below are secrets returned via get-secret-value. One from an environment account that does not have this issue and one from the account environment with the issue.

This is from an environment where there is no issue with DMS special characters.

2024-05-02T15:13:52.1728326Z {
2024-05-02T15:13:52.1729228Z     "ARN": "arn:aws:secretsmanager:us-east-1:acct:secret:<key_id>",
2024-05-02T15:13:52.1729608Z     "Name": "<key_id>",
2024-05-02T15:13:52.1729869Z     "VersionId": "terraform-20240418180941168400000001",
2024-05-02T15:13:52.1730903Z     "SecretString": "{\"username\": \"userx\", \"password\": \"a_password_i_can_login_with\", \"engine\": \"sqlserver\", \"host\": \"instance.us-east-1.rds.amazonaws.com\", \"port\": \"1433\", \"dbname\": \"adatabase\"}",
2024-05-02T15:13:52.1731228Z     "VersionStages": [
2024-05-02T15:13:52.1731443Z         "AWSCURRENT",
2024-05-02T15:13:52.1731633Z         "AWSPENDING"
2024-05-02T15:13:52.1731755Z     ],
2024-05-02T15:13:52.1731909Z     "CreatedDate": 1713463783.921
2024-05-02T15:13:52.1732540Z }

This is from an environment where all DMS tasks are failing:

2024-05-02T14:07:31.5451580Z {
2024-05-02T14:07:31.5454345Z     "ARN": "arn:aws:secretsmanager:us-east-1:acct:secret:<key_id>",
2024-05-02T14:07:31.5455381Z     "Name": "<key_id>",
2024-05-02T14:07:31.5456560Z     "VersionId": "dac2e53d-bc48-403f-9c4d-70e0da1ef311",
2024-05-02T14:07:31.5458479Z     "SecretString": "{\"username\": \"userx\", \"password\": \"a_password_i_can_login_with\", \"engine\": \"sqlserver\", \"host\": \"instance.us-east-1.rds.amazonaws.com\", \"port\": \"1433\", \"dbname\": \"adatabase\"}",
2024-05-02T14:07:31.5459334Z     "VersionStages": [
2024-05-02T14:07:31.5459944Z         "AWSCURRENT",
2024-05-02T14:07:31.5460498Z         "AWSPENDING"
2024-05-02T14:07:31.5460790Z     ],
2024-05-02T14:07:31.5461121Z     "CreatedDate": 1714657316.178
2024-05-02T14:07:31.5461424Z }