How do I troubleshoot high latency on my Application Load Balancer in Elastic Load Balancing?

Short description

The following are common causes of high latency on an Application Load Balancer:

• Network connectivity/congestion issues
• High memory (RAM) utilization on backend instances
• High CPU utilization on backend instances
• Incorrect web server configurations on backend instances
• Issues with web application dependencies that run on backend instances
• Large geographical distance between clients or on-premises targets and Application Load Balancer

Resolution

To troubleshoot high latency on your Application Load Balancer, complete the following steps:

1. Check for network connectivity issues. For more information, see Troubleshoot your Application Load Balancers.

2. Measure the first byte response and check for slow DNS resolution that might cause latency:

   curl -kso /dev/null -w "\n===============\n
   | DNS lookup: %{time_namelookup}\n
   | Connect: %{time_connect}\n
   | App connect: %{time_appconnect}\n
   | Pre-transfer: %{time_pretransfer}\n
   | Start transfer: %{time_starttransfer}\n
   | Total: %{time_total}\n
   | HTTP Code: %{http_code}\n===============\n" https://example.com/

   Example output:

   ===============
   | DNS lookup: 0.002346
   | Connect: 0.003080
   | App connect: 0.008422
   | Pre-transfer: 0.008587
   | Start transfer: 0.050238
   | Total: 0.057486
   | HTTP Code: 200
   ===============

   Note: Complete the preceding step with your Application Load Balancer. Then, complete the preceding step while you bypass your Application Load Balancer. This process helps you to isolate the component that causes latency. For more information, see curl.1 the manpage on the curl website.

3. Check the Average statistic of the Amazon CloudWatch TargetResponseTime metric for your Application Load Balancer. If the value is high, then you have an issue on the backend instances or the application dependency servers. For more information, see How do I troubleshoot an increase in the TargetResponseTime metric for an Application Load Balancer?

4. To identify the backend instances that have high latency, check the access log entries of your Application Load Balancer.

5. To identify backend instances with latency issues, check the target_processing_time for a longer than normal time period.

6. To confirm any issues with the Application Load Balancer, review the request_processing_time and response_processing_time fields for longer than normal time periods.

   Example log entry:
   Note: In the following example, the request_processing_time is the sixth value, the target_processing_time is the seventh value, and the response_processing_time is the eighth value.

   http 2024-04-01T22:23:00.765170Z app/example-loadbalancer/50dc6c495c0c9188
   192.168.131.39:2817 10.0.0.1:80 0.001 12.401 0.001 200 200 34 366
   "GET http://www.example.com:80/ HTTP/1.1" "curl/7.46.0" - -
   arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/example-targets/73e2d6bc24d8a067
   "Root=1-58337262-36d228ad5d99923122bbe354" "-" "-"
   0 2024-04-01T22:22:48.364000Z "forward" "-" "-" "10.0.0.1:80" "200" "-" "-"

   Note: The preceding example log entry indicates the request_processing_time is 0.001, the target_processing_time is 12.401, and the response_processing_time is 0.001. The target_processing_time value indicates a longer than normal time period and that the Application Load Balancer target has caused latency. For more information, see Syntax.

7. To identify high CPU utilization or spikes in CPU utilization, check the CloudWatch CPUUtilization metric of your backend instances. To resolve high CPU utilization, upgrade your instance to a larger instance type.

8. To review the Apache processes on your backend and check for memory issues, run the following command:

   watch -n 1 "echo -n 'Apache Processes: ' && ps -C apache2 --no-headers | wc -l && free -m"

   Example output:

   Every 1.0s: echo -;n 'Apache Processes: ' && ps -;C apache2 -;no-headers | wc -1 && free -;m
   Apache Processes: 27
             total     used     free     shared     buffers     cached
   Mem:      8204      7445     758      0          385         4567
   -/+ buffers/cache:  2402     5801
   Swap:     16383     189      16194

9. To check how many simultaneous requests that your instance can serve, view the MaxClient setting for the web servers on your backend instances. If your instance has an appropriate amount of memory and CPU utilization and still experiences high latency, increase the MaxClient value.

10. Compare the number of processes generated by Apache (httpd) with the MaxClient setting. If the number of Apache processes frequently reaches the MaxClient value, then increase the MaxClient value.

    Example command:

    [root@ip-192.0.2.0 conf]# ps aux | grep httpd | wc -l 15

    Example output:

    <IfModule prefork.c>StartServers         10
    MinSpareServers      5
    MaxSpareServers      10
    ServerLimit          15
    MaxClients           15
    MaxRequestsPerChild  4000
    </IfModule>

11. Check for dependencies on your backend instances that might cause latency issues. Dependencies include shared databases or external resources, such as Amazon Simple Storage Service (Amazon S3) buckets. Also, dependencies include external resource connections, such as network address translation (NAT) instances, remote web services, or proxy servers.

12. To identify performance bottlenecks on the server, use the following Linux tools:
    uptime -; Shows load averages to help identify the number of tasks (processes) that are waiting to run. On Linux systems, this number includes processes that are waiting to run on the CPU and processes blocked in uninterruptible I/O or disk I/O. Uptime provides a high-level view of the resource load (or demand) that require other tools to further analyze data. When Linux load averages increase, resources have a higher demand. To determine the resources that have a higher demand, use other metrics. Use mpstat -P ALL 1 for CPU utilization and top or pidstat 1 for per process CPU utilization.
    mpstat -P ALL 1 -; Shows CPU time breakdowns per CPU that you can use to check for imbalances. A single, hot CPU might indicate a single-threaded application.
    pidstat 1 -; Shows per process CPU utilization and prints a rolling summary that you can use to watch for patterns over time.
    dmesg | tail -; Shows the last 10 system messages. Use this tool to check for performance issues.
    iostat -xz 1 -; Shows the workload that's applied for block devices (disks) and the related performance.
    free -m -; Shows the amount of free memory. Make sure that the amount isn't near zero in size. If the amount is near zero, then you might experience higher disk I/O and decreased performance.
    sar -n DEV 1 -; Shows the network interface throughput (rxkB/s and txkB/s) as a measure of workload. Check if the limits have been reached.
    sar -n TCP and ETCP 1 -; Shows key TCP metrics that include active/s (locally-initiated TCP connections per second), passive/s (remotely-initiated TCP connections per second), and retrans/s (TCP retransmits per second).
    iftop -; Shows the connections between your server and a remote IP address that consume the most bandwidth. The n iftop tool is also available in a package with the same name on Red Hat and Debian-based distributions. For Red Hat-based distributions, look for n iftop in a third-party repository.