Instability in RDS MySQL db after increasing storage space

0

Hello,
Since 06-March-2020 our RDS instance (t2.micro) abruptly issues the message "Recovery of the DB instance has started. Recovery time will vary with the amount of data to be recovered" and proceeds to reboot. This is in contrast to our experience in the past SEVEN months where we didn't face any performance issue with RDS.

Our problem can be boiled down to these three parts:

(A)
On 6th of this month, our db instance reached the 20GB (free tier) limit on storage space. This prompted us to:
(1) increase storage size to 50 GB
(2) Enable elastic storage.
(3) Increase max_connections parameter value to 95 (from the default 66)

(B)
Soon we began to notice periodic instability with the db instance. Logs indicate this starts with a crash of the MySQL server. All these days this would be over in a couple minutes. However, today, this lasted for eight minutes.

(C)
We have noticed that our application server (a Java app) cannot "see" the newly restarted db, We usually stop the app and restart it. After this step things function smoothly the rest of the day.

We would like to know the precise reason for the repeated failures after the storage space was increased. We would also like to know if we need to configure the EC2 instance for it to "see" the recovered data instance.

dprasad
asked 4 years ago784 views
3 Answers
0

It's highly unlikely your problems are associated with the increase in storage size. That you increased max_connections is more of a hint to me as it suggests you are driving the t2.micro harder and perhaps beyond what its limited memory can support. Also remember you have the CPU burst balance to deal with. All database systems are subject to some level of instability if you push them too far beyond the resource limits, and I believe MySQL is one of the worst offenders. You might want to do some performance analysis to see what is going on.

On the connectivity front it sounds like the application does not properly handle connections dying. It should be getting an error and re-connecting.

HalTemp
answered 4 years ago
0

Thanks a lot! This clarified a couple things for sure. It was helpful.

We had reset max_connections parameter value to its default value soon after we posted the question. Therefore, yesterday was a better day, once again. There was a very brief period when our app maxed on DB connections, but it recovered immediately. It was such a relief!

The reconnecting logic could be tricky when the system is already saturated. We will figure this out.

dprasad
answered 4 years ago
0

This is just to inform that the RDS instance has been stable as it used to be in the past. All that we did was to reset max_connections parameter. Little did I know it would cause such stress, panic and havoc.

dprasad
answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions