Storage increase seems to be taking days

0

We have a production RDS (MySQL 5.6) that is running low on storage and wanted to see what the impact of increasing the storage would be.

We've brought up a backup of our production instance, set it to multi-AZ and waited until the write IOPS went to zero to before increasing the storage size from 2048GB to 3500GB. The read and write IOPS has since been hovering around 70-90 and the instance is marked as "modifying" for over 48 hrs. I can see that the new storage has been added tot he instance and I do have complete access to the instance. I'm concerned about potential load issues on production should we do this.

Any idea what's going on?

asked 3 years ago169 views
2 Answers
0

In most cases scaling storage occurs in a short amount of time, however for a small set of legacy volumes it is not possible to modify the volume in place. In these cases the duration of the scale storage depends on the size of the original volume, which is quite large in your case.
In your case I see you restored from your prod instance, this new instance you are testing is a physical copy of that prod instance.

The good part is that the instance is available for the duration of the scaling operation. And when this scaling operation completes, this instance will not require the legacy mechanism going forward, once complete any future scaling will use the faster method.

If you absolutly cannot run the scale storage on your prod instance, an alternative wouldbe to create a read replica, then scale the read replica and promote this to be the new prod db.

Hope that helps.

-Phil

AWS
MODERATOR
philaws
answered 3 years ago
0

Hi Phil,

So we did exactly this solution. We ran a few tests and everything went well so we waited until the holidays were over before running this on production. As you can likely see if you take a look, the replica lag is huge and growing slowly everyday. I've doubled the instance type and double checked all of the relevant flags and parameter group options I could think of to boost performance but can't seem to get anywhere.

Any ideas?

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions