EC2 Autoscaling group launching more instances than required
Hey everyone,
I have setup an ecs cluster with ec2 autoscaling group as capacity provider. When I deploy one service to my cluster. It launches 2 ec2 tasks(or service as there is just one task in service). And then later on kill one of the instance by scaling down. If I launch 2 tasks then it scales to 4 instances and then after that scales down after some time. Now the most problematic thing here is that as its launching instances more than required I am charged for minimum of one hour for each instance it launches. And this stuff is happening continously. And because termination protection is off it's sometimes deleting instances which were in use and so they get deployed again which again triggers to launch 4 instances making this run in an infinite loop. This would become pretty expensive for me if its not stopped at this point. I am clearly able to see that there are just 2 instances in use but still 2 more are there for no reason. Some information that might be needed -
- Capacity provider target capacity is set to 100%:
- And that capacity provider is being used as default capacity provider strategy:
Would be great if anyone can help me fix this. Thanks
- Newest
- Most votes
- Most comments
If managed termination protection is turned on when you create a capacity provider, the Auto Scaling group and each Amazon EC2 instance in the Auto Scaling group must have instance protection from scale in turned on.
If managed scaling is turned on when you create a capacity provider, the Auto Scaling group desired count can be set to 0. When managed scaling is turned on, Amazon ECS manages the scale-in and scale-out actions of the Auto Scaling group.
What have you set your Min, Required instances too in the ASG? Set as low as possible and let the CP manage onwards.
Can you TICK/check the above. This I think would behave better
Relevant content
- asked 5 months ago
- asked 2 years ago
AWS OFFICIALUpdated 4 months ago- AWS OFFICIALUpdated 2 years ago
Hey thanks for your response Gary. I have turned on manage scale in protection which fixed the issue of infinite loop. But that issue of scaling to 4 is still occuring. And it sometimes even gets more wild as on launching 4 tasks it scaled to 18 instances. Regarding your question of min required instances I set it to 0.
18 hmm. Are you getting targetgroup/health check failures or issues with Instance health status?
Are you using an Existing ASG or did you create from fresh for ECS?
To create an empty Auto Scaling group, set the desired count to zero. After you created the capacity provider and associated it with a cluster, you can then scale it out.
When you use the Amazon ECS console Create Cluster with the Amazon EC2 instances option under Infrastructure, Amazon ECS creates an Amazon EC2 Auto Scaling launch configuration and Auto Scaling group on your behalf as part of the AWS CloudFormation stack. They are prefixed with EC2ContainerService-<ClusterName>, which make them easy to identify. You can use the Auto Scaling group as a capacity provider for that cluster.
Hey There were no health check failures. I created a new asg actually that's the only asg because I never used asg's before using ecs. And autoscaling group only has one scaling policy that was given by ecs. Capture screenshot of that here -> https://ibb.co/BwQ5SN3 Cant upload an image here so shared a link of uploaded image.
And this btw -> https://ibb.co/d2PCpSj shows where it scaled to 18 from 6. And after that it got down to 4 after some mins.