Intermittent "AbortedOrigin" error, depends on Keep-alive setting

Question

I have a Cloudfront distribution connected to an AL2023 EC2 origin running Apache 2.4.62. The traffic includes POSTs as well as GETs. There are roughly 6000 POSTs per day, fairly evenly spread through the day, so there is one transaction every few seconds to a minute or so. The traffic is all HTTPS. The origin's CPU and network are seldom more than 1% busy.

Everything works great 99.9% of the time. About 0.1% of the POST transactions are failing with a 502 status. When this happens, the Cloudfront logs show the "x-edge-detailed-result-type" is "AbortedOrigin". There is no sign in the origin's logs of any matching transaction reaching the origin.

I disabled the WAF Shield security options (which was the main reason I chose to use Cloudfront for this service) but that made no difference.

Then I started playing around with the Keep-alive timeout setting. This defaults to 5s in both Cloudfront and Apache. I now have the keep-alive set to 60s in the Cloudfront origin settings, and 100s in the Apache httpd.conf, and that seems to have greatly reduced the error rate. With my traffic pattern, this would mean that a lot more traffic is using an existing connection, whereas with the default it would mostly be reconnecting. I think there is a race condition bug in the way Cloudfront and/or Apache handles a keep-alive connection timeouts.

I wonder if anyone else has seen this error? What is the best way to report a bug in AWS?

re:Post Agent · Answer

The issue you're experiencing with intermittent "AbortedOrigin" errors and 502 status codes is not uncommon when dealing with CloudFront distributions connected to EC2 instances running web servers like Apache. The fact that increasing the keep-alive timeout settings has reduced the error rate supports your theory about a potential race condition in connection handling.

When CloudFront forwards requests to the origin, it attempts to maintain persistent connections to improve performance. However, if the keep-alive timeout is too short, CloudFront might try to use a connection that the origin server has already closed, resulting in the "AbortedOrigin" error.

Your approach of increasing the keep-alive timeout on both CloudFront (to 60 seconds) and Apache (to 100 seconds) is a good strategy. This allows more requests to reuse existing connections, reducing the likelihood of encountering the race condition.

To further optimize your setup, consider the following:

1. Ensure that the origin response timeout in CloudFront is set appropriately. The default is 30 seconds, but you can increase it up to 60 seconds if needed.

2. Monitor your origin's health and performance closely. Even though CPU and network usage are low, there might be other factors affecting response times.

3. If possible, implement health checks on your origin to ensure it's responding correctly.

4. Consider enabling origin keep-alive in CloudFront, which can help maintain persistent connections more effectively.

As for reporting a potential bug to AWS, the best approach is to contact AWS Support. If you have an Enterprise or Business support plan, you can open a technical support case directly. For other support plans, you can use the AWS forums or reach out to AWS on social media platforms. When reporting, provide detailed information about your setup, the error logs, and the steps you've taken to troubleshoot.

While your current configuration seems to have improved the situation, continue monitoring the error rate. If you still see occasional issues, it may be worth exploring other factors such as network connectivity between CloudFront and your origin, or potential limitations in your EC2 instance type.  
**Sources**  
[Optimize high availability with CloudFront origin failover - Amazon CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html)  
[HTTP 504 status code (Gateway Timeout) - Amazon CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-504-gateway-timeout.html)  
[Cloudfront timeout to Application Load Balancer | AWS re:Post](https://repost.aws/questions/QU8hwBJby2SLWJFxly5Q_Y8w/cloudfront-timeout-to-application-load-balancer)

Intermittent "AbortedOrigin" error, depends on Keep-alive setting

Add your answer

Relevant content