Why am I seeing high latency with CloudFront cache hits, DNS delays, and slow OpenSearch in me-central-1

external tests show 300+ ms latency, including 200 ms from DNS

Is the DNS lookup to a custom domain or cloudfront.net domain?

If custom domain, are you hosting your DNS service on Route 53? For custom domain, is this a CNAME or Route 53 alias record to cloudfront.net? A R53 alias record is likely to be faster. DNS results are cached by recursive DNS servers and clients, which translates to shorter latencies for subsequent DNS queries.

Which DNS server are you using for lookup?

Below are 2 DNS lookups to a d1s7tw5t7fz258.cloudfront.net domain using Google (8.8.8.8) and Cloudflare (1.1.1.1) DNS servers. Note the 15 ms and 44 ms times.

% dig d1s7tw5t7fz258.cloudfront.net @8.8.8.8

; <<>> DiG 9.16.28 <<>> d1s7tw5t7fz258.cloudfront.net @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11208
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;d1s7tw5t7fz258.cloudfront.net. IN      A

;; AUTHORITY SECTION:
cloudfront.net.         60      IN      SOA     ns-418.awsdns-52.com. hostmaster.cloudfront.net. 1377556270 16384 2048 1048576 60

;; Query time: 15 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun Apr 06 13:17:32 Malay Peninsula Standard Time 2025
;; MSG SIZE  rcvd: 125


% dig d1s7tw5t7fz258.cloudfront.net @1.1.1.1

; <<>> DiG 9.16.28 <<>> d1s7tw5t7fz258.cloudfront.net @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59874
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;d1s7tw5t7fz258.cloudfront.net. IN      A

;; AUTHORITY SECTION:
cloudfront.net.         60      IN      SOA     ns-418.awsdns-52.com. hostmaster.cloudfront.net. 1377556270 16384 2048 1048576 60

;; Query time: 41 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sun Apr 06 13:17:55 Malay Peninsula Standard Time 2025

There are several factors contributing to the high latency you're experiencing with your CloudFront setup in the me-central-1 region. Let's break down the issues and potential solutions:

1. DNS and Connection Delays: The high DNS lookup times (100-150 ms) and connection delays are likely due to the geographical distance between your users and the CloudFront edge locations. CloudFront uses a global network of edge locations to cache and serve content, but the effectiveness can vary based on user location and edge location proximity. To reduce DNS lookup time, you could consider using Amazon Route 53 as your DNS service, which integrates well with CloudFront and can help optimize DNS resolution.

2. Cache Hit Latency: Even with cache hits, you're seeing 300+ ms latency. This could be due to the distance between the user and the nearest CloudFront edge location. To improve this, you might want to analyze your user base's geographical distribution and ensure you're using the most appropriate CloudFront price class that covers your users' locations.

3. ALB and ECS Fargate: Your Application Load Balancer (ALB) and ECS Fargate setup seems reasonable with the keep-alive and idle timeout settings. However, the high target_processing_time when the cache expires indicates that your application or database (OpenSearch) might be the bottleneck.

4. OpenSearch Performance: The perceived slower performance of OpenSearch in me-central-1 compared to ap-southeast-1 could be due to various factors such as cluster size, instance types, or data volume. It's worth investigating if there are any region-specific differences in the OpenSearch setup or if there are any network latency issues between your ECS tasks and the OpenSearch cluster.

5. Reducing Overall Latency:

• Use CloudFront Origin Shield to reduce the load on your origin and potentially improve cache hit ratios.
• Optimize your caching strategy in CloudFront to reduce the frequency of requests to your origin.
• Consider using Amazon ElastiCache in front of OpenSearch to cache frequent queries and reduce load on the OpenSearch cluster.
• Analyze and optimize your application code and database queries to reduce processing time.
• If possible, replicate your OpenSearch data across multiple regions and use latency-based routing to direct users to the nearest copy of your data.

6. Monitoring and Analysis: Use AWS CloudWatch to monitor the performance of your ALB, ECS tasks, and OpenSearch cluster. This can help identify bottlenecks and areas for optimization. Additionally, use CloudFront's real-time logs and standard logs to analyze cache performance and identify patterns in user requests.

Remember that latency can be affected by many factors, including network conditions, server load, and application efficiency. A holistic approach considering all components of your architecture is often necessary to achieve significant latency reductions.
Sources
Caching and availability - Amazon CloudFront
Evolving the architecture with Amazon CloudFront - Hosting Static Websites on AWS
Troubleshoot latency on calls or requests to Amazon ECS tasks | AWS re:Post
Speeding up your Amazon S3-based website using Amazon CloudFront - Hosting Static Websites on AWS