I have read most, all?, of the AWS troubleshooting guides on RDS/Aurora and we tried a bunch of things. We have a cluster with one reader and one writer. About two days ago they failed over and flipped automatically. Not sure why, not even sure it's related, but it happened and we didn't notice because things kept humming along. This morning, however, our apps, and IDEs started receiving too many connection errors when trying to connect. We could connect directly to the read instance, but couldn't write. Rebooted both, waited for the writer instance to recover, and same issue. Failed over to flip the instances, no real change. Flipped back, no change. Rebooted the writer again, no change. As we finally gave up and started restoring a snapshot, write connections just resumed on their own. There was about a half hour gap between the previous reboot and no additional event logs that would indicate it just took a while to reboot or failover.
This feels like there was an AWS issue, but of course I don't see anything mentioned in the health dashboard. Our max connection param was set to the default for this version of MySQL and worked out to 90 for the db.t2.medium's we're running on. According to Cloudwatch we have never seen much more than about 15, so it seems odd that we were even receiving too many connection errors.
It would be great to know what changed to break things and then how they were resolved because right now I have a working instance but no real clue as to what may have changed this morning. Any ideas?
AWS OFFICIALUpdated a year ago
If you have confirmed that this is not a problem with too many connections on the application side, the only thing you can do is report this issue to AWS Support. In the case of such situations, I create new environment from backup and I leave the old instance for problem analysis, for AWS Support. I met with such situation that the problem stopped appearing on new environment and I could continue my work with new instance. Unfortunately AWS Support wasn't able to find root cause then.