EC2 Autoscaling group launching more instances than required
Hey everyone,
I have setup an ecs cluster with ec2 autoscaling group as capacity provider. When I deploy one service to my cluster. It launches 2 ec2 tasks(or service as there is just one task in service). And then later on kill one of the instance by scaling down. If I launch 2 tasks then it scales to 4 instances and then after that scales down after some time. Now the most problematic thing here is that as its launching instances more than required I am charged for minimum of one hour for each instance it launches. And this stuff is happening continously. And because termination protection is off it's sometimes deleting instances which were in use and so they get deployed again which again triggers to launch 4 instances making this run in an infinite loop. This would become pretty expensive for me if its not stopped at this point. I am clearly able to see that there are just 2 instances in use but still 2 more are there for no reason. Some information that might be needed -
- Capacity provider target capacity is set to 100%:
- And that capacity provider is being used as default capacity provider strategy:
Would be great if anyone can help me fix this. Thanks
- Newest
- Most votes
- Most comments
Here since, the managed scaling is turned on, Amazon ECS manages the scale-in and scale-out actions of the ASG and, if termination protection is enabled, ECS will not terminate the protected Task while scaling in. I'd highly recommend you to look into ECS cluster Auto-scaling blog for in depth understanding.
That makes sense but still not sure why its launching 4 ec2 instances. That is the main problem.
Hey thanks for your suggestions scale in protection fixed infinite loop now the thing that is left is to fix ecs launching more ec2 instances than required.
Hi,
Did you envision ECS Fargate, which is fully serverless, for your use case? See https://aws.amazon.com/fargate/
Then, Fargate will manage the underlying servers. You will only pay for the container compute resources that you use.
This post may help you further optimize: https://medium.com/@spei/cost-optimization-quick-wins-on-amazon-ecs-with-aws-fargate-15948d73123b
Best,
Didier
Hey Didier,
I actually know about this but according to our workloads and requirements EC2 was a better option. So I really don't want to switch to fargate. I just want to fix the current setup.
Thanks, Tushar
Relevant content
- asked 2 years ago
- asked 5 months ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 4 months ago
Is minimumScalingStepSize set on your capacity provider? If so, that would lead to this 'always scaling by 2' behavior you're mentioning https://docs.aws.amazon.com/cli/latest/reference/ecs/create-capacity-provider.html
As a side note, just an FYI that most instances have per-second billing now, not hourly
Hey Shahad yes I did set minimumScalingStepSize but that is set to 1 which is the default value. So would it still be an issue? Also thanks for telling up that per second billing thing was very worried of costs lol. Although its somewhat fine now but still if I can find a solution for the problem, then that would be great.