MySQL RDS Replica Promotion Stuck in "Backing-Up" State

0

Good evening,

We are in the middle of upgrading one of our RDS instances from MySQL 5.6 to MySQL 5.7. To minimize downtime, we took the path of creating a replica, upgrading that replica to 5.7, and then promoting it to the primary. The issue is after we selected to promote the database, its stuck in "Backing Up" state and has been for the past hour. Before the promotion, we placed the primary instance into "Read Only" mode via a parameter group setting.

There is no indication that anything is happening, and the log files aren't updating either. The DB instance itself is still responsive to traffic.

Is this normal? AWS documentation said the promotion process would take "a few minutes". Snapshots/backups on our current production database only take about 2 minutes to complete...

1 Answer
1
Accepted Answer

backing-up state means the database is available but it is taking a snapshot. So maybe the promotion is already done and the database is available for use. The replicas normally do not have backups enabled and it is mandatory for primaries to have them, so once a replica gets promoted it starts taking a backup, and this operation can take time as the first backup needs to copy the entire disk snapshot. backing-up instances are usable but not modifiable in AWS console.

After the first snapshot, the rest of them are faster, like in your production database, because consecutive snapshots only copy the disk differences between snapshots.

I've found that sometimes enabling backups and taking a manual snapshot of the replica meant to be promoted before the promotion speeds up any automated backup created later, but YMMV.

theist
answered 2 years ago
  • Thanks a ton for the context and how you generally approach these things! I'll definitely give that a try on the next one as I have several to do. The job did end up completing, it just took about an hour or hour 20. Otherwise it was a fairly seamless process! I've definitely accepted your answer. Thanks again!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions