CloudFront Slow in One Region

Hi,

In the case a given CloudFront PoP, becomes slower than the others, you can try to use Route53 latency-based routing: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-latency.html

When configured properly, Route53 shall detect that answers from a given Pop get slow and route them to another Pop where latency remains better. An example of this is https://aws.amazon.com/blogs/networking-and-content-delivery/latency-based-routing-leveraging-amazon-cloudfront-for-a-multi-region-active-active-architecture/

This blog post has a double use of Route53: the one useful / similar to you own use case is the one labeled with (1).

Best,

Didier

Based on the information provided, it appears you're experiencing a localized issue with a specific CloudFront Point of Presence (PoP) in one region. This is an unusual situation, as CloudFront is designed to improve content delivery speed and reliability across different regions.

Here are some potential reasons and steps to investigate:

1. PoP Performance Issue: The specific X-Amz-Cf-Pop you mentioned could be experiencing performance problems. This might explain why the issue is isolated to one region and why bypassing CloudFront resolves it.

2. Regional Edge Cache: There might be an issue with the Regional Edge Cache serving that particular PoP. Regional edge caches are larger than individual PoPs and help keep less popular content closer to viewers.

3. Origin Shield Configuration: If you're using Origin Shield, ensure it's configured correctly and in the nearest region to your origin. An incorrectly configured Origin Shield could lead to increased latency.

4. Cache Settings: Review your cache policies for the affected content. Ensure that static content is set to be cached (CachingOptimized) and dynamic content is set appropriately (CachingDisabled if necessary).

5. Origin Response Time: Although you've tested directly to the Load Balancer without issues, double-check the origin response times when requests come through CloudFront. There could be a network or routing issue between the specific PoP and your origin.

6. CloudFront Logs: Enable detailed logging for your CloudFront distribution and analyze the logs for the affected requests. Look for any patterns or error codes that might explain the delay.

7. AWS Support: Given that this issue started suddenly without changes on your end and persists even after WAF was disabled, it might be worth contacting AWS Support. They can investigate if there are any known issues with the specific PoP or regional infrastructure.

8. Temporary Workaround: If the issue is critical and persistent, you could consider creating a new CloudFront distribution or updating your DNS settings to route traffic from the affected region to a different PoP temporarily.

Remember, CloudFront typically maintains persistent connections with origin servers to fetch objects quickly. The fact that this issue is isolated to one region and PoP suggests it might be an infrastructure-related problem on AWS's end rather than a configuration issue on your side.

Continue monitoring the situation and gather as much data as possible (logs, specific request examples, timings) to assist in troubleshooting, whether you're working on it internally or with AWS Support.
Sources
How CloudFront delivers content - Amazon CloudFront
Cloudfront content download speed is too slow. | AWS re:Post
Key Features of a Content Delivery Network – Performance, Security – Amazon CloudFront

Are the responses CloudFront is receiving from the ALB cacheable? Requests from the CloudFront edge location to an ALB should time out in 30-60 seconds, so if there were a connectivity issue between them, the client should receive an HTTP 504 timeout response rather than a valid response after 130 seconds.

Where are you measuring the 130-second response time? Do you see it in CloudFront's access logs, response headers, or are you measuring it from the client side?

If you can reproduce the issue in a regular Chromium-based web browser (like Chrome), could you run a trace with Developer tools on the Network tab? It would give a breakdown of the different stages of executing the request, including everything from DNS requests through connection establishment to receiving the first and last bytes of the response. Tick the "Disable caching" checkbox to bypass client-side caching in Chromium/Chrome.