How do I resolve Aurora MySQL-Compatible blue/green deployment upgrade errors?

Short description

If you use a blue/green deployment to upgrade an Aurora MySQL-Compatible database, then your deployment might show Invalid configuration for the state of the upgrade. To find more information about the issue, check the Logs & Events section of the writer instance for the following message:

"Failed to provision due to upgrade incompatibilities."

Resolution

Prepare for the blue/green deployment

To reduce issues with your blue/green deployment, complete the following best practices to prepare for the deployment:

• Review the Aurora MySQL version compatibility matrix to make sure that you can upgrade your current version to the target version with a blue/green deployment. For more information, see Checking Aurora MySQL version numbers.
• Assess the ability of your application to handle brief interruptions during cut-over. To assist with the cut-over process, implement connection retry logic in your application code. For more information, see General best practices for blue/green deployments.
• Review all configuration settings and test your application workload to make sure that your client or application is compatible with the target version. For more information, see Planning a major version upgrade for an Aurora MySQL cluster.
• Create a backup of your current database to make sure that you can roll back if needed.

Execute the blue/green deployment

During your deployment, take the following actions:

• Use Amazon CloudWatch metrics, Amazon EventBridge, and CloudWatch Logs to monitor the upgrade progress.
• If you upgrade to a later minor version within the same major version, then use the in-place upgrade feature on the green cluster. This feature allows you to reach the latest patch version before you complete the cut-over process.
• After you confirm that the green environment is ready and the green cluster meets all switchover guardrails, complete cut-over during a planned maintenance window. For more information, see Switchover guardrails for blue/green deployments.

If the switchover stalls or slows down during cut-over, then check the error logs for specific error messages or bottlenecks. You can specify a switchover timeout period between 30 seconds and 3,600 seconds (1 hour). If the switchover takes longer than the specified duration, then Aurora rolls back changes and doesn't change either environment.

Note: The default timeout period is 300 seconds (5 minutes).

Review configurations after the upgrade

After you complete the cluster upgrade, verify that all database functions, stored procedures, and application queries work as expected. Monitor performance metrics for 24–48 hours after the upgrade, including peak hours, to make sure that there aren't any unexpected performance effects.

Resolve the incompatibility error

If your current database configuration uses settings or features that aren't compatible with the target database, then you get the following error message:

"Failed to provision due to upgrade incompatibilities. See https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.mysql80-upgrade-procedure.html#AuroraMySQL.mysql80-upgrade-troubleshooting to resolve these issues, then delete and recreate this Blue/Green Deployment."

To resolve this error, complete the following steps:

1. Review the upgrade-prechecks.log file in your Aurora MySQL-Compatible database.
2. Address any compatibility issues that you find in the log file. For more information, see Summary of Community MySQL upgrade prechecks and Summary of Aurora MySQL upgrade prechecks.
3. Delete the blue/green deployment.
4. Recreate the blue/green deployment.

Resolve the parameter compatibility error

If your green cluster is in an incompatible-parameters status, then existing cluster parameters or instance parameter values aren't compatible with the target version.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To resolve this error, complete the following steps:

1. Reset the cluster parameter groups and instance parameter values.
2. To promote the green cluster to a standalone cluster, run the following promote-read-replica-db-cluster AWS CLI command:

   aws rds promote-read-replica-db-cluster \
       --db-cluster-identifier GREEN-CLUSTER-ID

   Note: Replace GREEN-CLUSTER-ID with your green DB cluster identifier.
3. Delete the DB cluster and DB instance.
4. Review the compatibility of the existing cluster's parameter values with the target version. Or, switch to default values on the blue cluster. Then, change the parameter values one at a time during cut-over to the green cluster.
5. After you confirm the parameters, create a new blue/green environment.

Resolve a stalled upgrade

Your green cluster might stall during your upgrade because of target version incompatibility. Long-running transactions on the blue cluster might also block the upgrade process.

To resolve this issue, complete the following steps:

1. Check whether the upgraded version supports the target version for direct upgrade.
2. If the version isn't supported, then revert your changes and delete the blue/green deployment.
3. Upgrade to the recommended intermediate version. Then, upgrade to the final version.

You can also monitor your blue cluster for long-running transactions. If there are long-running transactions, then complete or end the transactions. Then, retry the upgrade process.

Related information

How do I upgrade my Amazon Aurora MySQL DB cluster to a new version?

Upgrade to Amazon Aurora MySQL version 3 (with MySQL 8.0 compatibility)

Performing major version upgrades for Amazon Aurora MySQL with minimum downtime