Route 53 Failover debugging

Question

So we had an incident with our newly implemented failover strategy. Our new setup has R53 pointing to an ALB as the primary DNS entry and a CloudFront distribution pointing to an S3 bucket static page as the Secondary. Both are using Aliased entries and the corresponding AWS healthchecks (not R53 defined healthchecks). The setup was stable for ~2days and then did a failover to the secondary. Looking at Cloudwatch and our saved logs for the incident there isn't any clear indication as to what might have caused the failover, only data showing the corresponding shift in traffic. No spike/blip in 5XXs or 4XXs requests, that would indicate that the ALB or the corresponding target instances were unhealthy. So I am a bit baffled.

* Is there anything specific we should look for?
* Is there an R53 log or monitor that shows the failover? 
* Is/are there additional logs we should be capturing?
  * Like alias healthcheck data

NARRAVULA MUNI SAI TEJA · Answer

Hello Sigmonia,

I can help you debug the Route 53 failover incident you experienced.

**Route 53 Health Check Logs:** Route 53 might have clues even with ALB/CloudFront checks. Check logs for your ALB/CloudFront aliases around the failover time: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html.

**ALB & CloudFront Logs:** Look for access log anomalies in CloudWatch around the failover that might have caused Route 53 to suspect unhealthy resources.

**Examine ALB and CloudFront Logs:** Enable and review these logs for errors or unusual patterns.

Route 53 Failover debugging

Add your answer

Relevant content