db cluster modification said success, but it didn't work

I understand your frustration with the upgrade process not working as expected. Let's go through a few points to help clarify the situation and provide some suggestions:

1. First, it's important to note that you're attempting a major version upgrade from Aurora MySQL version 2 (compatible with MySQL 5.7) to Aurora MySQL version 3 (compatible with MySQL 8.0). This is a significant upgrade that requires more careful planning and execution than a minor version upgrade.

2. For major version upgrades, the process is not as straightforward as simply modifying the engine version in the cluster settings. The "successfully modified" message you saw might have been related to initiating the upgrade process rather than completing it.

3. Major version upgrades can take considerable time to complete, especially for larger databases. The "upgrading" status you observed is normal, and it may take a while before the new version is fully applied.

4. If the cluster is still showing the old version after some time, it's possible that the upgrade encountered an issue or was rolled back due to compatibility problems.

To address this situation, here are some suggestions:

1. Check the Events tab for your DB cluster in the RDS console. This may provide more detailed information about what happened during the upgrade attempt.

2. Consider using a blue/green deployment for the major version upgrade. This method allows you to create a new cluster with the desired version, replicate data from the old cluster, and then switch over when ready. This approach minimizes downtime and provides a safer upgrade path.

3. If you decide to try the upgrade again, make sure to review the upgrade prerequisites and compatibility issues between your current version and the target version. Resolve any potential conflicts before attempting the upgrade.

4. You may want to try upgrading to an intermediate version first (e.g., the latest 2.x version) before jumping to 3.x, as this might help identify and resolve any compatibility issues more easily.

5. If problems persist, it would be advisable to contact AWS Support for personalized assistance, especially given the approaching deadline for avoiding extra support fees.

Remember, major version upgrades are complex operations that require careful planning and testing. Always ensure you have recent backups and consider testing the upgrade process in a non-production environment first.
Sources
Upgrading the minor version or patch level of an Aurora MySQL DB cluster - Amazon Aurora
Upgrading Amazon Aurora MySQL DB clusters - Amazon Aurora
Upgrading Aurora MySQL by modifying the engine version - Amazon Aurora

Hi, Please follow the steps below to quickly address the issue and ensure your upgrade is successful.

1. Check the Version

   • Console Refresh: The AWS Console might not reflect the latest state. Try refreshing the page. • Verify via CLI or API: Use the AWS CLI to confirm the engine version and ensure the upgrade was applied successfully.

aws rds describe-db-clusters --db-cluster-identifier <your-cluster-id> --query "DBClusters[0].EngineVersion" Ensure the version returned is “8.0”.

2. Investigate Potential Causes

   1. Check Parameter Group Settings: • The upgrade may not apply if the associated Parameter Group isn’t aligned with the new engine version (8.0). • Make sure the correct parameter group is applied, or create a new one suited for the upgraded version.
   2. Check Replica Instances: • Confirm that both Writer and Reader nodes were upgraded. • If any reader instance is still on the older version, the upgrade might not reflect properly.
   3. Check for Pending Reboots: • After the upgrade, some instances may require a reboot to apply changes. • Use the console or CLI to check for instances pending a reboot.

aws rds reboot-db-instance --db-instance-identifier <your-instance-id>

3. Review Logs and Events

   • Check the RDS Event Logs for any errors or messages during the upgrade. Go to the AWS Console > RDS > Event Logs and review relevant events.

4. Attempt a Manual Upgrade via CLI

   • If the problem persists, try upgrading the cluster manually through the CLI.

aws rds modify-db-cluster \ --db-cluster-identifier <your-cluster-id> \ --engine-version "8.0.mysql.aurora.3.05.2" \ --apply-immediately I hope this helps you resolve the problem quickly!

In the end the upgrade problem was resolved once we found the "upgrade-precheck" log which showed the errors to resolve before upgrading. There were several tables that had datetime in an older format that had to be fixed - and an unused view. The key was finding this log amongst all of the other logs. It would be awesome if AWS could say specifically to look for the log rather than just having a green status saying "upgrade complete". Like a red status saying "upgrade failed - check the upgrade-prechecks log for more information".