AWS ECS Service Auto Scaling IN(decreasing) is not working.

0

Hi,

I have configured AWS managed cluster auto scaling and service auto scaling based on target tracking policies. When the threshold of 80% was met, a new task was launched. However, there were no available resources to run the new task, causing the cluster capacity to reach 100%. At that point, a new instance was launched. This process worked perfectly — the scale-out action functioned as expected.

The Service Auto Scaling Group (ASG) settings are:

Memory: Tracking ECSServiceAverageMemoryUtilization at 80% (scale-in enabled)

Now, for the past hour, memory utilization has been consistently below the threshold value, and the associated alarm remains in the "In Alarm" state. However, the task has not been stopped or terminated.

I need help in scaling in the service.

  • Is it the service or the cluster not scaling in? If its the Service, then focus on the Service scaling policy(s) and alarms vs the ASG.

1 Answer
0

The issue you're experiencing with ECS Service Auto Scaling not scaling in (decreasing) as expected is a common concern. There are several factors that could be contributing to this behavior:

  1. Cooldown periods: Auto Scaling has built-in cooldown periods to prevent rapid fluctuations. Make sure enough time has passed since the last scaling activity. HUMAN EDIT: scale-out cooldown doesn't block scale-in actions in Application AutoScaling

  2. Multiple scaling policies: If you have multiple scaling policies (e.g., for CPU and memory), scale-in only occurs when all policies agree it's necessary. Check if you have other policies that might be preventing scale-in.

  3. Minimum capacity: Ensure that your current task count isn't already at the minimum capacity set for your service.

  4. Task protection: If you're using task-level protection, it could prevent tasks from being terminated during scale-in events.

  5. Deployment in progress: During ECS deployments, scale-in processes are typically turned off to maintain service stability.

To address this issue:

  1. Review your Auto Scaling settings, including cooldown periods and minimum capacity.

  2. If you have multiple scaling policies, consider using a single target tracking policy that combines CPU and memory metrics. HUMAN EDIT: Better option would be to disable scale-in on one policy if you think scale-in should only be tracked on a single metric. HOWEVER, this is generally not advised for CPU + Memory. The goal of this behavior is to only scale-in when BOTH are low at the same time, otherwise you end up with oscillation where CPU wants to scale-out; then Memory scales-in; CPU scales-out, etc

  3. Check for any active deployments or task protections that might be interfering with scale-in.

  4. Ensure that the "Turn off scale-in" option is not enabled for your scaling policy.

  5. You can check why Application AutoScaling chose to ignore the low usage alarm via the not-scaled-reasons field

If the issue persists, you may need to manually adjust your task count or review your scaling policy thresholds. Remember that Auto Scaling is designed to be conservative with scale-in actions to maintain service stability, so some delay in scaling in is normal.
Sources
ECS Service auto scaling policy conflict the one to other policy | AWS re:Post
Amazon ECS EC2-backed auto scale-out and scale-in | AWS re:Post
ECS scale-in no SIGTERM signal | AWS re:Post

profile picture
answered 16 days ago
AWS
EXPERT
revised 9 days ago
AWS
EXPERT
revised 9 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions