RDS Instance stuck in rebooting (no connections)

0

Hello all!

Earlier today, one of our RDS instances ran out of space. This was unexpected and very sudden considering we freed up a ton of space by truncating stale data but after doing so, it suddenly consumed all available space, ran out and shut itself down.

As a temporary measure, I've allocated an additional TB of storage. RDS modified successfully but then went into storage-optimization mode. During that state, the instance was not accessible - it showed to be online (i.e. the was CPU usage, IOPS count was in the 1500s) but it showed zero connections. After roughly 6 hours of that, I've decided to reboot the instance since it has did not appear to be going anywhere - and this is where we are now... stuck in Rebooting state. I am also unable to log into the instance from normal external sources.

Wondering where I go from here? This is an instance with failover however right now the only options I have in the actions menu is to delete it.

Thanks in advance!

asked 2 years ago963 views
1 Answer
-1

Hi, I understand your RDS instance is stuck in rebooting and not accepting connections and you'd like to know how this can be mitigated. From your response, I see that your RDS ran out of storage. Please note that when RDS runs out of storage, connections will not be possible. You then increased storage and RDS instance was in "storage optimization" mode. As per document here [1], "storage optimization" does not prevent connection. It only prevents further storage modification actions within the next 6 hours. The connection issue could be due to the persistence in low storage available. You then tried to reboot the RDS but it is stuck in rebooting. Please note that the reboot process will not complete when the RDS does not have storage available. As you mentioned that the only option you see is "Delete", you can either perform the either of the below mitigative steps:

  • Contact AWS Premium Support [2] in order to get engineers an internal look at the RDS
  • Or Delete the RDS. Ensure to check if you have a manual backup and restore it. I hope the above information is helpful.

Reference links:

[1]: Working with storage for Amazon RDS DB instances - Increasing DB instance storage capacity - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.ModifyingExisting [2]: https://aws.amazon.com/premiumsupport/plans/

Winnie
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions