Unexpected AZ change in SingleAZ installations

AWS typically moves database instances to different Availability Zones (AZs) during events such as AZ maintenance, infrastructure issues, or AWS-managed failover scenarios. Even in a Single-AZ deployment, AWS can change the AZ under certain circumstances to maintain availability and resilience.

Let’s break down your questions:

1. Why does AWS do this?

AWS may change the Availability Zone of your database instances in the following cases:

**Infrastructure failures: **If there are issues with the underlying hardware or network infrastructure in a specific AZ, AWS might automatically move instances to another AZ to ensure availability.

**Maintenance: **AWS sometimes performs maintenance that requires an instance to be moved to a different AZ.

**High availability measures: **In Single-AZ instances, if AWS detects issues that could potentially affect the instance's availability, it may decide to relocate the instance to another AZ for safety.

This is done to ensure that your database remains available and does not experience prolonged downtime.

2. How is this done without downtime and without DNS changes?

AWS uses several techniques to minimize or eliminate downtime during AZ moves:

Elastic IP and DNS management: Even in Single-AZ setups, AWS uses internal mechanisms like Elastic IP addresses, Route 53 DNS, and internal DNS names that allow the underlying instance to change AZs without changing the public endpoint or requiring manual intervention.

Automatic failover and recovery mechanisms: For instance, in RDS, AWS automatically handles the transition, keeping the same endpoint for database connections, so clients do not need to update DNS records or configuration.

Typically, this results in a short, automatic failover without a need for manual DNS management.

3. Can this be avoided?

It’s generally not possible to entirely prevent AWS from moving instances if there is a strong enough reason (e.g., AZ failure, maintenance). However, some ways to reduce the likelihood of unexpected AZ changes include:

**Use Multi-AZ Deployments: **Multi-AZ configurations are designed to handle failovers better by keeping a standby instance in a different AZ. AWS automatically handles failovers with minimal impact.

Avoid Single-AZ Replicas: In your setup, with Single-AZ replicas in different AZs, AWS can still move them for maintenance. Using a Multi-AZ setup for your master and replicas would ensure more stability in AZ configurations.

4. Can we restrict the change to an AZ different from the master’s?

No, AWS doesn’t provide a way to restrict which AZ an instance can move to. It manages failovers and instance moves based on factors such as AZ availability, infrastructure health, and capacity, without user control over specific AZ placement. The instance might end up in the same AZ as the master if that’s deemed the best available option by AWS.

5. How can we be alerted when this happens?

To stay informed when AWS changes the AZ of your instances, you can do the following:

Amazon RDS Event Subscriptions: You can configure RDS event subscriptions to receive notifications via Amazon SNS when certain events happen, such as failovers, AZ changes, or instance moves. This will help alert you when AWS relocates a database instance.

Go to the RDS console, select Event Subscriptions, and subscribe to relevant events (e.g., instance failovers, maintenance).

CloudWatch Alarms: You can monitor instance metrics through Amazon CloudWatch and set up alarms for changes in your instances. This way, you will receive alerts if AWS performs failovers or AZ changes.

Enhanced Monitoring: Enabling enhanced monitoring for RDS instances provides more granular visibility into system-level metrics, which may help detect changes like AZ shifts.

Summary:

AWS might move instances due to maintenance, AZ failures, or other infrastructure issues.

These moves are managed internally to minimize downtime, using mechanisms like Elastic IPs or internal DNS routing.

You can’t prevent AZ changes entirely, nor can you restrict which AZ an instance is moved to.

Set up notifications through RDS event subscriptions or CloudWatch to be alerted when AZ changes occur.

Moving to a Multi-AZ setup would provide more predictable failovers and separation between your master and replica.