How do I resolve the error: "The following parameters are not defined for the specified group: xxx, xxx" when I upgrade the engine version of my RDS cluster using CloudFormation?

Short description

When you upgrade an Amazon RDS DB cluster or instance that uses a custom parameter group, you receive an error if you update

• The EngineVersion property for AWS::RDS::DBCluster or AWS::RDS::DBInstance
  -or-
• The Family property for DBClusterParameterGroup or DBParameterGroup

For example, in the following template, updating the EngineVersion property from 5.7.37 to 8.0.28 in the DBInstance resource causes the stack update to fail. The stack update also fails when you update the Family property from MySQL5.7 to MySQL8.0 in the DBParameterGroup resource.

Parameters:
  DBName:
    Default: MyDatabase
    Description: The database name
    Type: String
  DBUser:
    NoEcho: 'true'
    Description: The database admin account username
    Type: String
  DBPassword:
    NoEcho: 'true'
    Description: The database admin account password
    Type: String
Resources:
  MyDB:
    Type: 'AWS::RDS::DBInstance'
    Properties:
      DBName: !Ref DBName
      AllocatedStorage: '5'
      DBInstanceClass: db.t2.small
      Engine: MySQL
      EngineVersion: 5.7.37 
      MasterUsername: !Ref DBUser
      MasterUserPassword: !Ref DBPassword
      DBParameterGroupName: !Ref MyRDSParamGroup
      AllowMajorVersionUpgrade: true
  MyRDSParamGroup:
    Type: 'AWS::RDS::DBParameterGroup'
    Properties:
      Family: MySQL5.7
      Description: CloudFormation Sample Database Parameter Group
      Parameters:
        autocommit: '1'
        general_log: '1'
        old_passwords: '0'

Note: There are other scenarios that can cause this error message. The steps in the following Resolution section apply only to the preceding scenario.

Resolution

The following sets of steps are two ways that you can resolve the error: "The following parameters are not defined for the specified group: xxx, xxx."

Note: Applying a parameter group to a DB instance might initiate an instance reboot. During the reboot, there is a database outage.

1. Add a new ParameterGroup resource to the stack template with the new Family value and keep the old ParameterGroup resource in the template.
2. Reference the new ParameterGroup resource in AWS::RDS::DBCluster or AWS::RDS::DBInstance, and update the EngineVersion property to the new version.
   Note: If you're performing a major version upgrade, then you must set the AllowMajorVersionUpgrade property to true.
3. Update the stack with the updated template.
4. After the stack is updated, remove the previous ParameterGroup resource from the stack template.

-or-

1. Change the LogicalResourceID of the ParameterGroup resource.
2. Reference the new LogicalResourceID in the AWS::RDS::DBCluster or AWS::RDS::DBInstance.