AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
ECS Service Auto Scaling Continuously Triggering Due to Missing Service-Level CloudWatch Metrics
Question Content Problem Summary: My ECS service auto scaling is continuously triggering scaling attempts every 15-30 minutes, generating "AlreadyAtDesiredCapacity" and "AlreadyAtMinCapacity" messages, despite low CPU, memory, and request loads. The service never actually scales up from 1 instance.
Service Details:
- Region: us-west-2
- Platform: Fargate
- Current Configuration: Min: 1, Max: 3, Desired: 1
- Task Definition: 256 CPU units, 512MB memory
- Auto Scaling Policies:
- CPU Utilization: Target 75%
- Memory Utilization: Target 75%
- ALB Request Count: Target 150 requests per target
Root Cause Identified (from Amazon Q): Service-level CloudWatch metrics (with dimensions ServiceName and ClusterName) stopped emitting data at a specific timestamp, while task-level metrics continue working fine and are visible in the ECS console "Health and metrics" tab.
Current Situation:
- Task is running normally
- Task-level CPU/Memory metrics: ✅ Working (visible in ECS console)
- Service-level CPU/Memory metrics: ❌ Stopped publishing to CloudWatch
- Auto Scaling: Continuously attempting evaluation but can't retrieve metrics
Observed Behavior:
- Auto scaling attempts occur every 15-30 minutes
- Messages like "AlreadyAtDesiredCapacity" and "AlreadyAtMinCapacity" appear repeatedly
- Service stays at minimum capacity (1 task) but never scales up
- Actual resource utilization is low (CPU ~10%, Memory ~20%, very few requests)
Questions:
- Why would service-level CloudWatch metrics stop publishing while task-level metrics continue working?
- Is this a known issue with ECS CloudWatch metrics aggregation?
- What's the difference between task-level and service-level metric publishing in ECS?
- How can I restore service-level metrics without affecting the running task?
- Should I delete and recreate the scaling policies, or wait for metrics to resume?
What I've Verified:
- Task is healthy and running normally
- Task-level metrics are publishing correctly to CloudWatch
- Auto scaling policy configuration is correct
- No recent changes to task definition or service configuration
- CloudWatch logs show normal application behavior
Technical Details:
- The service uses Fargate with awsvpc network mode
- CloudWatch logging is configured properly
- Task has been running stable for over 24 hours
- Issue appears to be specifically with service-level metric aggregation
Has anyone experienced similar issues where ECS service-level CloudWatch metrics stop publishing while task-level metrics continue normally? Any insights on what might cause this metric publishing gap or how to restore it would be greatly appreciated.
- Newest
- Most votes
- Most comments
I deleted all auto scaling, "reset" it then recreated, all related alarms recreated, it looks normal now. Keep monitoring, thanks guys!
Service level metrics is basically combining the metrics across the tasks within a service. If you have only 1 task, then it might appear similar. You don't need to do anything enable the service level metrics specifically. It should be enabled. Based on your screenshots, it seems it's working.
You are seeing scale in events but since you have reached the minimum, it cannot scale in any further. Nothing unusual.
Hi SHAJAM, thanks for your input!
I have 4 similar ECS like that and setting same scaling policy, while the other 3 are running well, this only one keep "attempting" hourly and daily, this is strange, a good infra system should not have that.
You will see it keep Error, success.... no one should feel ok and safe with that kind of system, it's not usual at all!
I used AmazonQ and it could not help, that's why I try to reach out to others to see if any solution, many thanks!
answered 9 months ago
Relevant content
asked 3 years ago
- AWS OFFICIALUpdated 2 years ago
