Skip to content

Amazon RDS stuck in some reboot loop with lost connectivity

0

Hi,

See the attached image. Our Aurora serverless V2 entered into some odd state during 30 minutes of todays production. Sporadically we lost connection to it from our application and it appears it entered into some odd restart loop. The problem is the error logs are empty.

Enter image description here

We did a force reboot and since then everything has been fine, however as we're not quite sure what caused this we need to find the reason to avoid it happening again or to be able to mitigate the risk.

Earlier the same day we increased our ACCU:s from 24 to 48 due to a spike in load. But that was hours before this incident.

Any ideas?

asked a year ago322 views

2 Answers
0

Given that you increased the ACUs (capacity units) earlier that day, it might have contributed. Rapid scaling or a spike in load can sometimes cause internal state inconsistencies or trigger failovers in Aurora Serverless. The zero-downtime restart attempts indicate the system was trying to stay available during these events, but something kept causing it to restart again.

Since the error logs are empty, I’d recommend enabling Enhanced Monitoring and Performance Insights if not already, to get more granular metrics about resource usage and potential bottlenecks during the incident. Also, review any recent parameter group changes or configuration updates that might affect stability. Then check your application’s connection pooling and retry logic, bursty reconnects can put additional pressure during scaling.

If possible, set up alerts to catch such restart loops early.

answered a year ago

0

Based on my experience managing Aurora Serverless v2 in production, intermittent connection drops and unexplained restart behavior like this can sometimes be caused by internal scaling events or resource contention, even if the logs appear clean.

It’s definitely unusual that the error logs are empty during the incident. I’ve seen this happen when Aurora enters a transient fault or memory-related state where the internal health check triggers restarts but doesn’t surface detailed diagnostics unless enhanced monitoring or Performance Insights is enabled.

The fact that a force reboot resolved the issue suggests the cluster might have entered a partial fault state that didn’t recover properly on its own. I've seen similar issues happen during heavy load transitions or compute reallocation events.

You mentioned increasing ACCUs from 24 to 48 earlier that day. Although this occurred hours before, I’ve learned that ACCU scaling changes can trigger internal maintenance or stabilization processes, especially if the cluster was already under stress from production traffic. It’s worth checking if there were any background maintenance events or scaling activities around that time using CloudWatch logs or the RDS Events section.

For future mitigation, I recommend:

  • Enabling enhanced monitoring and Performance Insights (if not already)
  • Reviewing the Amazon RDS Events log for subtle failover or compute instability messages.
  • Keeping a close eye on connection spikes, memory pressure, or transaction queue depth using CloudWatch metrics.

If this issue persists, raising a support ticket may be your best option to have internal logs reviewed. I’ve had cases where only AWS Support could access low-level diagnostics not exposed to the console.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.