How to detect a Route 53 failover?

Hi,

This article is a solution to achieve what you want: https://medium.com/@opsbridge/route-53-dns-failover-with-lambda-healthchecks-in-private-subnet-e488b5a62e0c

The lambda monitoring checks which route is taken and can raise alerts on failover

Best,

Didier

Hello,

To detect a Route 53 failover:

If you're using Route 53 health checks You can set up CloudWatch to monitor the health check status. If a failover happens, CloudWatch will show the health check failing, and you can create alerts based on that.

If you're only using "Evaluate Target Health" AWS automatically manages failover based on the status of the target (like an EC2 or ELB). To spot this, you can track the health of the target service in CloudWatch.

https://medium.com/@opsbridge/route-53-dns-failover-with-lambda-healthchecks-in-private-subnet-e488b5a62e0c

Detecting a Route 53 failover involves monitoring the health of your resources and Route 53 configuration. Here are methods to detect failovers with and without Route 53 health checks:

Using Route 53 Health Checks and CloudWatch Metrics

1. Create a Route 53 health check for your resource (e.g., endpoint, instance).
2. Configure the health check to send data to CloudWatch.
3. Create a CloudWatch metric (e.g., HealthCheckStatus) to track the health check state.
4. Set up a CloudWatch alarm to trigger when the health check fails (e.g., HealthCheckStatus = UNHEALTHY).
5. Integrate the alarm with AWS services like SNS, Lambda, or EC2 Auto Recovery.

CloudWatch Metrics for Route 53 Health Checks

• HealthCheckStatus (UNHEALTHY/HEALTHY)
• HealthCheckPercentageHealthy (0-100%)
• HealthCheckCount (number of health checks)

Without Route 53 Health Checks (Evaluate Target Health)

1. Monitor your resource's health using CloudWatch metrics (e.g., CPU utilization, latency).
2. Create a CloudWatch alarm to trigger when the resource's health metric exceeds a threshold.
3. Use AWS Config to monitor Route 53 resource record sets and detect changes.
4. Integrate AWS Config with CloudWatch Events to trigger actions on resource changes.

CloudWatch Events for Route 53

• "ChangeResourceRecordSets" event (detects changes to resource record sets)
• "Route 53 Resource Health" event (detects resource health changes)

Additional Methods

1. Amazon CloudWatch Logs: Monitor Route 53 API calls and errors.
2. AWS X-Ray: Analyze application performance and errors.
3. Third-party monitoring tools (e.g., Datadog, New Relic).

Best Practices

1. Implement redundant health checks and monitoring.
2. Set up notifications and automated failover processes.
3. Regularly test and validate failover scenarios.

To detect a Route 53 failover in your scenario, where you have alias records with Evaluate Target Health enabled but no explicit Route 53 health checks, you can use the following approach:

CloudWatch Metrics for VPC Endpoint and S3: Since you're using a VPC endpoint as the primary target and an S3 bucket as the secondary, you can monitor the health and traffic of these resources using CloudWatch metrics.

For VPC Endpoint:

Monitor the "AWS/PrivateLinkEndpoints" namespace Key metrics: "ActiveConnections", "NewConnections", "ProcessedBytes" For S3:

Monitor the "AWS/S3" namespace Key metrics: "NumberOfRequests", "4xxErrors", "5xxErrors" Set up CloudWatch Alarms: Create alarms based on these metrics to detect unusual patterns or drops in traffic to your primary VPC endpoint, and increases in traffic to your S3 bucket.

Use AWS Lambda: Create a Lambda function that periodically checks the status of your VPC endpoint and S3 bucket. This function can use the AWS SDK to query the health status of these resources.

Amazon EventBridge: Set up EventBridge rules to trigger based on state changes in your VPC endpoint or S3 bucket. These can be used to invoke your Lambda function or send notifications directly.

Custom Application Logs: If possible, implement logging in your application to record which endpoint it's connecting to. This can help you identify when traffic shifts from the VPC endpoint to the S3 bucket.

DNS Queries: You can periodically perform DNS queries for your Route 53 record and log the results. A change in the resolved IP address can indicate a failover.

Best Practices and Considerations:

Implement a combination of the above methods for more reliable detection.

Set appropriate thresholds for your CloudWatch alarms to avoid false positives.

Consider implementing a "cool-down" period in your detection logic to account for brief fluctuations that may not represent a true failover.

Ensure your Lambda functions and any custom scripts have the necessary IAM permissions to access the required resources and metrics.

Regularly test your failover detection mechanism to ensure it works as expected.

Consider using AWS SNS (Simple Notification Service) to send notifications when a potential failover is detected.

Remember that Evaluate Target Health for alias records relies on the health checking mechanisms of the target service (in this case, VPC endpoints and S3). Ensure you understand how these services report their health to Route 53.

Keep in mind that there might be a slight delay between when a failover occurs and when it's detected due to DNS TTL and propagation times.

By implementing these methods, you should be able to effectively detect and be notified of failovers in your Route 53 setup, even without using explicit Route 53 health checks.