スキップしてコンテンツを表示

Amazon RDS stuck in some reboot loop with lost connectivity

0

Hi,

See the attached image. Our Aurora serverless V2 entered into some odd state during 30 minutes of todays production. Sporadically we lost connection to it from our application and it appears it entered into some odd restart loop. The problem is the error logs are empty.

Enter image description here

We did a force reboot and since then everything has been fine, however as we're not quite sure what caused this we need to find the reason to avoid it happening again or to be able to mitigate the risk.

Earlier the same day we increased our ACCU:s from 24 to 48 due to a spike in load. But that was hours before this incident.

Any ideas?

質問済み 1年前334ビュー

2回答
0

Given that you increased the ACUs (capacity units) earlier that day, it might have contributed. Rapid scaling or a spike in load can sometimes cause internal state inconsistencies or trigger failovers in Aurora Serverless. The zero-downtime restart attempts indicate the system was trying to stay available during these events, but something kept causing it to restart again.

Since the error logs are empty, I’d recommend enabling Enhanced Monitoring and Performance Insights if not already, to get more granular metrics about resource usage and potential bottlenecks during the incident. Also, review any recent parameter group changes or configuration updates that might affect stability. Then check your application’s connection pooling and retry logic, bursty reconnects can put additional pressure during scaling.

If possible, set up alerts to catch such restart loops early.

回答済み 1年前

0

Based on my experience managing Aurora Serverless v2 in production, intermittent connection drops and unexplained restart behavior like this can sometimes be caused by internal scaling events or resource contention, even if the logs appear clean.

It’s definitely unusual that the error logs are empty during the incident. I’ve seen this happen when Aurora enters a transient fault or memory-related state where the internal health check triggers restarts but doesn’t surface detailed diagnostics unless enhanced monitoring or Performance Insights is enabled.

The fact that a force reboot resolved the issue suggests the cluster might have entered a partial fault state that didn’t recover properly on its own. I've seen similar issues happen during heavy load transitions or compute reallocation events.

You mentioned increasing ACCUs from 24 to 48 earlier that day. Although this occurred hours before, I’ve learned that ACCU scaling changes can trigger internal maintenance or stabilization processes, especially if the cluster was already under stress from production traffic. It’s worth checking if there were any background maintenance events or scaling activities around that time using CloudWatch logs or the RDS Events section.

For future mitigation, I recommend:

  • Enabling enhanced monitoring and Performance Insights (if not already)
  • Reviewing the Amazon RDS Events log for subtle failover or compute instability messages.
  • Keeping a close eye on connection spikes, memory pressure, or transaction queue depth using CloudWatch metrics.

If this issue persists, raising a support ticket may be your best option to have internal logs reviewed. I’ve had cases where only AWS Support could access low-level diagnostics not exposed to the console.

回答済み 1年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

関連するコンテンツ