How do I troubleshoot high response times in ECS?

3 minute read
1

I want to troubleshoot high response times and latencies with my Amazon Elastic Container Service (Amazon ECS) tasks.

Resolution

Check your Amazon ECS service-level metrics

Higher CPU and memory utilization can cause high response latencies from your Amazon ECS tasks. Use Amazon CloudWatch metrics to view service-level CPU and memory utilization. If you activated CloudWatch Container Insights, then check container-level CPU and memory utilization. If you didn't activate Container Insights, then see How do I configure CloudWatch Container Insights to monitor my Amazon ECS tasks and container instance?

If your application's containers must perform CPU-intensive workloads, then upgrade to a larger task size that includes more virtual CPU (vCPU) and memory. Or, turn on Application Auto Scaling.

Amazon ECS tasks in Amazon EC2 launch type

Check the Amazon Elastic Cloud Compute (Amazon EC2) instance's CPU and memory utilization of your Amazon ECS tasks that use the Amazon EC2 launch type.

Use CloudWatch metrics to monitor network performance of your Amazon EC2 instance. If the instance network bandwidth exceeds the available instance baseline bandwidth, then use a larger instance type with high baseline bandwidth.

Use the CPUUtilization CloudWatch metric to monitor instance CPU utilization. Also, log in to your instance to check the memory utilization. If CPU or memory utilization is high, then add more instances to the cluster to distribute tasks across multiple instances. Or, change the instance to a larger instance type.

Check the application logs to determine whether there's a high response latency from a downstream service. It's a best practice to activate AWS X-Ray to troubleshoot latency issues.

Measure response latency

If the Amazon ECS service is behind an Application Load Balancer, then use the TargetResponseTime CloudWatch metric to measure the response time. This metric provides the average response time for requests that are routed through the Application Load Balancer. If the value is high, then there might be delays in the backend application or infrastructure.

To measure response latency directly from the container, log in to the application container. For Amazon EC2 launch type tasks, run the Docker exec -it container id command to log in to the container. For tasks that use the Fargate launch type, use ECS exec to log in to the container.

Then, run the following command to measure response latency:

curl -kso /dev/null -w "\n===============\n
| Connect: %{time_connect}\n
| App connect: %{time_appconnect}\n
| Pre-transfer: %{time_pretransfer}\n
| Time to First Byte (TTFB): %{time_starttransfer}\n
| Total: %{time_total}\n
| HTTP Code: %{http_code}\n===============\n" http://localhost:port/endpoint

Note: Replace the port with your container port and endpoint with your endpoint.

In the preceding command's output, the Time to First Byte (TTFB) field shows the time that it takes for the first byte to arrive. The output also includes the server processing time. If the processing time is high, then the application is taking longer to process the request.

Related information

How do I troubleshoot high CPU utilization on an Amazon ECS task on AWS Fargate?

How can I monitor high memory utilization for Amazon ECS tasks on Fargate?

How do I calculate the total network usage for my EC2 instances and determine which instance used the most bandwidth within a date range?

AWS OFFICIAL
AWS OFFICIALUpdated 21 days ago