Skip to content

RDS Blue / Green issues

0

I'm trying to do a Blue / Green changeover but it's failing with the error:

Switchover from DB cluster asamazon2-cluster to asamazon2-cluster-green-xxxx was canceled due to external replication on asamazon2-cluster. Stop replication from an external database to asamazon2-cluster before you switch over.

If I do slow slave status on the Blue database I get:

Slave_IO_State: Master_Host: 221.121.140.xxx Master_User: xxx Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin-changelog.000016 Read_Master_Log_Pos: 904410367 Relay_Log_File: relaylog.205102 Relay_Log_Pos: 4 .........

This database has not be a slave for years as this is our main production database.

Have tried "CALL mysstop_replication;", but get the message " Slave is already stopped or may not be configured. Run SHOW SLAVE STATUS\G"

A response on Stackexchange suggested using CALL mysql.rds_reset_external_master; but that made the database hang and I had to reboot it.

I have since deleted that deployment and have tried again with a new one, but it completely failed to create a replica and AWS sent this:

"You tried to create Amazon Aurora cluster 'asamazon2-cluster-green-vy3nda' in the 'AP-SOUTHEAST-2' region. The operation couldn't succeed due to an issue during database instance creation. Because the current replica underwent a host replacement, the original symptom that caused the issue cannot be identified. To help us diagnose the issue, we would request you to delete this binlog replica cluster and create a new one. If the issue persists when creating the new replica we will be able to identify the symptom and apply the necessary mitigations."

Deleted that and have tried creating another Blue / Green environment, but 3 hours later it's still provisioning.

I know they're 2 seperate issues, but help for either of them would be appreciated.

3 Answers
0

I am not sure why rds_reset_external_master would hang your database, but you need to let that succeed in order for the output from 'show replica status' to no longer show coordinates to an external source database.

AWS
MODERATOR
answered 3 years ago
  • I'm not sure either. Have decided to go with an "in place" upgrade - have tested it on a clone of our current database and downtime is acceptable. The upgrade to 5.7 also seems to solve the issue being stuck as a slave.

0

I believe that you need to remove the configuration to the external source from the blue instance.

CALL mysql.rds_reset_external_master();
AWS
MODERATOR
answered 2 years ago
  • Peplica has been reset with this procedure. But Blue/Green Deployment does not make sense. We want to stop replication and when the upgrade is completed successfully, we want it to continue where it left off.

    Note: After the call of this procedure, it gets the same error as well.

0

Hi Team, I wanted to test Blue/Green Deployment for the stage environment but I got an error like "Switchover from DB cluster master (blue) cluster to green cluster was canceled due to external replication on the master cluster. Stop replication from an external database to the master cluster before you switch over." during the switchover process. After my controls with some of the procedures (SHOW REPLICA STATUS, SHOW SLAVE STATUS), the replication state looks stopped. How can I solve this issue and upgrade the MySQL version to 8.0 for the Master Cluster?

Procedure: "CALL mysql.rds_stop_replication;" 
Before the procedure call
Slave_IO_State: Waiting for the source to send the event.
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Slave_SQL_Running_State: Replica has read all relay logs; waiting for more updates.
After the procedure call
'Replica is down or disabled'

Slave_IO_State: Null
Slave_IO_Running: No
Slave_SQL_Running: No
Slave_SQL_Running_State: Null

Replica_IO_State:Null
Replica_IO_Running: No
Replica_SQL_Running: No
Replica_SQL_Running_State: Null
You can find the Database structure below:
Master Cluster (blue) -> Engine version: 5.7.mysql_aurora.2.11.2
Master DB Writer Instance (blue)
Master DB Reader Instance (blue)

Blue/Green Deployment 
Green Cluster -> Engine version:8.0.mysql_aurora.3.05.0
Green DB Writer Instance
Green DB Reader Instance
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.