Unexpected Aurora RDS Failover Promoted Smaller Reader to Writer and Demoted Writer to Reader

Question

Hello AWS Team,

We recently encountered an unexpected behavior in our Amazon Aurora RDS (MYSQL-compatible) production environment.

Our Aurora cluster consists of:
* **Writer instance**: db.r6g.4xlarge
* **Reader instance**: db.t3.medium

At one point, an automatic cross-AZ failover occurred, and AWS promoted the smaller reader instance (db.t3.medium) to become the new writer, while the original writer (db.r6g.4xlarge) was demoted to a reader.

This caused significant performance issues, as the newly promoted writer (db.t3.medium) was not capable of handling our production write load. We did not manually trigger this failover — it was done automatically by the system.

We would like to understand:

1. What triggered this failover? Were there any issues with the original writer that caused it to be demoted?
2. How can we prevent failovers to instances with insufficient capacity? Is there a way to control which instances are eligible to be promoted?
3. What are the best practices for configuring Aurora RDS clusters to ensure failovers do not impact performance in production environments?

We would appreciate any guidance or recommendations to better manage failover scenarios and maintain consistent performance.

Manvitha Potluri · Accepted Answer

Example for above:

**Use failover priority tiers.**
Aurora supports failover tiers, which control the order in which replicas are considered for promotion.

**Here’s how it works:**
Instances are assigned tier values from 0 (highest) to 15 (lowest)
During failover, Aurora promotes the lowest-numbered tier that's healthy
If multiple readers are in the same tier, Aurora picks based on instance availability and health

**In your case:**
Your db.r6g.4xlarge reader (if you had one) should be Tier 0
Your t3.medium reader should be Tier 1 or higher

Manvitha Potluri · Answer

Hey, what you ran into is something we’ve seen happen in Aurora clusters when failover priorities aren’t explicitly configured.

Aurora doesn't look at instance size during failover; it just promotes the first “healthy” reader it finds, unless you set up failover tiers. So your t3.medium reader got promoted because it was considered available, even though it’s undersized for your production write load.

Configure failover tiers
You can prevent this by assigning failover priorities to your readers:
Assign your high-capacity reader (same size as writer) to Tier 0
Assign smaller or test replicas (like t3.medium) to higher tiers like Tier 15

This tells Aurora to only promote eligible, production-ready replicas.

Example AWS CLI command:

aws rds modify-db-instance \
  --db-instance-identifier your-reader-id \
  --promotion-tier 15
You can also set this via the AWS Console under the “Failover priority” setting when modifying a replica.

Best practices going forward:
Always have at least one writer-sized reader in a different AZ.
Use failover tiers to avoid surprises.
Monitor for failover events using RDS Event Subscriptions.
If you need stronger availability across regions, consider Aurora Global Database.

This fix is permanent; once you assign tiers, Aurora will follow that order every time. Let me know if you need help updating the tier settings or testing a manual failover safely.

Dhvani S · Answer

Hi, Aurora failover prioritization can be critical for maintaining optimal performance during failover events. When failover priorities aren't explicitly configured, Aurora simply promotes the first available reader, regardless of instance size, which can lead to performance issues if an undersized instance (like a t3.medium) becomes the writer. To prevent this, configure failover tiers by assigning your high-capacity reader (matching the writer's size) to Tier 0 and smaller/test replicas to higher tiers (like Tier 15). This ensures only production-ready replicas are promoted. Use the AWS CLI command aws rds modify-db-instance --db-instance-identifier your-reader-id --promotion-tier 15 or configure via AWS Console. Best practices include maintaining at least one writer-sized reader in a different AZ, using failover tiers consistently, monitoring failover events through RDS Event Subscriptions, and considering Aurora Global Database for cross-region availability. Once configured, these tier settings permanently guide Aurora's failover behavior, ensuring reliable performance during failover events.

EXPERT

Dhvani S

answered a year ago

Unexpected Aurora RDS Failover Promoted Smaller Reader to Writer and Demoted Writer to Reader

Add your answer

Relevant content