How do I monitor high resource usage in ECS tasks and services?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Monitor key metrics

Use Amazon CloudWatch to track ECS service-level metrics. For more detailed resource usage information, turn on Container Insights or Container Insights with enhanced observability and check the CPU utilization and memory usage metrics.

Analyze CloudWatch logs

Run a sample query to analyze your logs for high resource usage, Out of Memory (OOM) events, or application errors. Optimize the task definition based on the issues that you find. For more information, see Analyzing log data with CloudWatch Logs Insights.

To check memory utilization, run the following query:

stats avg(MemoryUtilized) by bin (30m) as period, TaskDefinitionFamily, TaskDefinitionRevision
| filter Type = "Task" | sort period desc, TaskDefinitionFamily | limit 10

Troubleshoot unexpected task stops

If your ECS task unexpectedly stops, then use the DescribeTasks API to check stoppedReason and the container's exitCode to determine the cause. For more information, see Why is my Amazon ECS task stopped?

Use ECS Exec

Use ECS Exec from the AWS CLI to run commands inside your containers to view resource usage information.

Troubleshoot potential issues

For memory leaks, check application logs and processes to locate the memory leak source.

For high CPU usage, increase task CPU allocation or upgrade to a larger instance type.

For disk space issues, run the following command to identify files that are larger than 50 MB:

find / -type f -size +50000k -exec ls -lh {} \; 2>/dev/null

To increase your disk space, see How can I increase disk space for my Amazon ECS container on AWS Fargate?

Perform tests and optimize resource allocation

To determine the optimal resource allocation for your applications, use tools to perform tests. Then, update your task definitions and service configurations based on the results. Modify CPU and memory capacity in your task definitions based on your usage patterns. Use Application Auto Scaling to efficiently manage changing workloads.

Check instance capacity

Make sure that your container instances have sufficient resources for your tasks. Use capacity providers to automatically adjust your cluster capacity.

Set up CloudWatch alarms

Create alarms for metrics, such as CPU utilization, memory usage, and application error rates. Configure notifications to alert you when thresholds are exceeded.

Related information

How do I mount an Amazon EFS file system on an Amazon ECS container or task that runs on AWS Fargate?

Container Insights with enhanced observability now available in Amazon ECS

Amazon ECS CloudWatch metrics

Monitor Amazon ECS using CloudWatch