AWS Builder Center: Learn, Build and Connect with builders in the AWS community

AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.

How do I troubleshoot CPU and memory spikes that cause unexpected scaling in Amazon ECS?

6 minute read

Content level: Advanced

A guide to troubleshoot and prevent unexpected Application Auto Scaling events due to resource spikes with ECS Services

Short Description

My Amazon Elastic Container Service (Amazon ECS) tasks are experiencing CPU or memory spikes that trigger unexpected auto scaling events. I want to identify the root cause and configure target tracking scaling policies to handle resource spikes appropriately.

CPU and memory spikes in Amazon ECS tasks can cause unexpected scale-out events for the following reasons:
Health checks aren't accurately determining application health, so unhealthy tasks continue running while consuming excessive resources.
Target tracking thresholds are too low for your workload, causing premature scaling.
You aren't tracking ALBRequestCountPerTarget alongside CPU/memory metrics, resulting in scaling decisions that don't account for actual traffic patterns.
JVM heap memory settings use fixed sizes instead of percentages, causing memory pressure when task sizes change.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI . Also, make sure that you're using the most recent AWS CLI version.

Tasks appear healthy while CPU/memory spikes during auto scaling

If your service uses container health checks, Elastic Load Balancing (ELB) health checks, or both, ensure that the health check endpoint reflects actual application health rather than just reachability. For example, an endpoint at /health that always returns HTTP 200 confirms the container is reachable, but doesn't validate whether the application can serve traffic. A downstream dependency failure would go undetected. A more effective health check executes a functional path, such as querying a database or verifying connectivity to dependent services, to confirm the application can actually handle requests.

Tasks are failing health checks and stopping while CPU/memory is spiking during Service Autoscaling

During task startup, dependencies such as reading from a database, queuing service, or a storage location can hold up additional processing of incoming requests. An initial spike in resource utilization is expected, but you can configure your ECS service to handle startup latency and improve application availability.

Grace Period

Set a health check grace period to allow tasks time to satisfy all software dependencies before the load balancer routes traffic from the target group to newly running tasks. This prevents premature health check failures during initialization.

Minimum healthy percent and deployment configuration

During a deployment, should the minimumHealthyPercent allow for tasks to be stopped prior to new tasks being available and healthy, incoming traffic to remaining tasks may exceed their processing or memory. This causes resource spikes that trigger further health check failures. To avoid this:

Balance task availability against traffic demand for each ECS service.
Ensure minimumHealthyPercent and maximumPercent allow new tasks to start and pass health checks before old tasks are stopped. Align auto scaling cooldown periods with your deployment duration to avoid reactive scaling during rollouts. This is discussed in more detail in the Improving Service Autoscaling behavior section, below.

Improving Service Autoscaling behavior

To prevent CPU and memory spikes from causing unexpected scaling, benchmark your scaling metrics and consider configuring target tracking policies.

Use ALBRequestCountPerTarget alongside resource metrics

If your application is fronted by a load balancer, you can also use the ALBRequestCountPerTarget to scale your service. ALBRequestCountPerTarget measures the average number of requests each task receives. Unlike CPU or memory, which can spike due to application-level issues (garbage collection, connection pool exhaustion, cold starts), request count directly reflects actual traffic demand.

Create separate target tracking policies for each metric such as CPU, Memory and ALBRequestCountPerTarget. When you configure multiple target tracking policies on a single ECS service, Application Auto Scaling uses the following logic:

Scale-out: If any policy's metric breaches its target, the service scales out. The metric under the most pressure triggers the scaling action.
Scale-in: The service scales in only when all policies agree that utilization is below target.

This means that the most-utilized resource drives scale-out decisions, while scale-in is conservative and requires all metrics to be within acceptable range. This prevents a scenario where low CPU triggers scale-in while request count is still high.

Benchmarking your target values

To set appropriate targets:

Load test your service in a staging environment and record the CPU utilization, memory utilization, and ALBRequestCountPerTarget at your desired response-time threshold (SLO).
Set each target to the metric value observed when the service is handling its optimal load (for example, 60% CPU, 70% memory, 1000 requests per target).
Avoid setting targets too low. For example, a CPU target of 20% causes premature scale-out on minor fluctuations and a target of 60–70% provides headroom without over-provisioning.

Note: ALBRequestCountPerTarget is not supported for services using the blue/green deployment type.

For more information, see Use a target metric to scale Amazon ECS services.

JVM heap memory is lower than task size or container resource size

When migrating Java applications to containers, developers often carry over hard-coded JVM heap settings from VM-based deployments. A fixed -Xmx value can undersize the heap relative to available container memory, causing frequent garbage collection that drives up CPU utilization. Although you can allocate higher memory and CPU in the ECS task definition, the JVM cannot use the additional resources if its heap is capped by a hard-coded value.

To avoid this, use -XX:MaxRAMPercentage (commonly set to 75%) instead of -Xmx. This allows the heap to scale automatically when you change the container's memory allocation. You can also set -XX:InitialRAMPercentage to control initial heap size at startup. For a detailed walkthrough of JVM memory and CPU configuration in containers, see JVM memory, CPU, and classpath best practices for Java containers on AWS.