Cloudformation-based update of RDS DBSnapshotIdentifier fails with "failed validation constraint for keyword [pattern]"

1

We are attempting to update the snapshot identifier of a MySQL DBInstance via Cloudformation (via CDK, but that does not appear relevant to the error). The current diff is as follows:

Resources
[~] AWS::RDS::DBInstance AbcDatabase AbcDatabase6789 replace
 └─ [~] DBSnapshotIdentifier (requires replacement)
     ├─ [-] arn:aws:rds:eu-central-1:12345:snapshot:downsync20220125030153
     └─ [+] arn:aws:rds:eu-central-1:12345:snapshot:downsync20220126030040

Unfortunately, this unexpectedly started to fail last night:

a-stack | 0/8 | 3:18:53 AM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | a-stack User Initiated
a-stack | 0/8 | 3:19:22 AM | UPDATE_FAILED        | AWS::RDS::DBInstance       | AbcDatabase (AbcDatabase6789) Properties validation failed for resource AbcDatabase6789 with message:
#: #: only 1 subschema matches out of 2
#/DBSnapshotIdentifier: failed validation constraint for keyword [pattern]
a-stack | 0/8 | 3:19:22 AM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack | a-stack The following resource(s) failed to update: [AbcDatabase6789]. 
a-stack | 1/8 | 3:19:26 AM | UPDATE_COMPLETE      | AWS::RDS::DBInstance       | AbcDatabase (AbcDatabase6789) 
a-stack | 2/8 | 3:19:27 AM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | a-stack 
a-stack | 3/8 | 3:19:28 AM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | a-stack 

This is an operation that we perform every night. The last successful update was at about 2022-01-25 03:16:25 +01:00, the first failing attempt at 2022-01-25 03:18:53 +01:00.

Looking at the CloudTrail logs, Cloudformation does not attempt to call any RDS APIs during the failing update. (For example: We do not see a RestoreDBInstanceFromDBSnapshot event which is normally triggered during successful runs by the Cloudformation service user.) We therefore conclude, that the failure is within Cloudformation.

Based on this other issue, the message looks like a JSON spec validation error, but in the relevant CloudFormationResourceSpecification.zip we could not find any validation rules in aws-rds-dbinstance.json ... especially no 'pattern' disallowing the specified snapshot ARN.

Does anyone have an idea what is going wrong? Or even better: What hidden restrictions the snapshot identifier suddenly has to conform to? (We can adjust the snapshot names, if that would help.)

Example of a successful update:

Resources
[~] AWS::RDS::DBInstance AbcDatabase AbcDatabase6789 replace
 └─ [~] DBSnapshotIdentifier (requires replacement)
     ├─ [-] arn:aws:rds:eu-central-1:12345:snapshot:downsync20220124031945
     └─ [+] arn:aws:rds:eu-central-1:12345:snapshot:downsync20220125030153

resulting in

a-stack | 0/8 | 3:15:55 AM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | a-stack User Initiated
a-stack | 0/8 | 3:16:24 AM | UPDATE_IN_PROGRESS   | AWS::RDS::DBInstance       | AbcDatabase (AbcDatabase6789) Requested update requires the creation of a new physical resource; hence creating one.
a-stack | 0/8 | 3:16:25 AM | UPDATE_IN_PROGRESS   | AWS::RDS::DBInstance       | AbcDatabase (AbcDatabase6789) Resource creation Initiated
0/8 Currently in progress: a-stack, AbcDatabase6789
a-stack | 1/8 | 3:31:34 AM | UPDATE_COMPLETE      | AWS::RDS::DBInstance       | AbcDatabase (AbcDatabase6789) 
[...]
a-stack | 7/8 | 3:32:44 AM | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack | a-stack 
[...]
a-stack | 3/8 | 3:32:48 AM | DELETE_IN_PROGRESS   | AWS::RDS::DBInstance       | AbcDatabase (AbcDatabase6789) 
3/8 Currently in progress: a-stack, AbcDatabase6789
[...]
a-stack | 3/8 | 3:39:56 AM | DELETE_COMPLETE      | AWS::RDS::DBInstance       | AbcDatabase (AbcDatabase6789) 
a-stack | 4/8 | 3:39:57 AM | UPDATE_COMPLETE      | AWS::CloudFormation::Stack | a-stack 
  • The issue resolved itself tonight, so it appears to have been a transient issue somewhere within CloudFormation.

  • The problem has returned today. The error message is unchanged: #: #: only 1 subschema matches out of 2 #/DBSnapshotIdentifier: failed validation constraint for keyword [pattern]

  • ... aaand it works again. Obviously flaky. Very annoying! :-(

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions