ECS: Autoscaling scale down the wrong instance
Hello
We have a web application which start on demand algorithms. When a user start an algorithm, it starts a task in an ECS Cluster, which has a capacity provider configured to provide an EC2 to run the algorithm.
Here I have 2 issues :
-
When ECS detect that it needs to provide an EC2, 2 servers are scaled while only 1 task is started
-
As we can see in the screenshot, after 15 min the capacity provider detects that it needs only 1 EC2, so it scale down one EC2, but sometimes, it kills the server on which the algorithm is running. Logs from amazon-ecs-agent on tail -f during the execution of the task :
level=info time=2022-11-17T13:20:40Z msg="Managed task at steady state" task="e4874c5693044ed08acbeb311c4919a5" knownStatus="RUNNING"
level=info time=2022-11-17T13:20:40Z msg="Managed task at steady state" task="e4874c5693044ed08acbeb311c4919a5" knownStatus="RUNNING"
Connection to 35.180.71.XXX closed by remote host.
Is there a way to provide only 1 EC2 when we start a task ?
Is ECS the right way to do these actions ?
Thank you, Valentin
- 최신
- 최다 투표
- 가장 많은 댓글
Here are some ideas how you could manage these on-demand jobs better:
- If you are just running the container to execute one time algorithm then AWS Batch might be better way for you to launch and manage those containers.
- If you don't want implement AWS Batch and you like your current ECS setup. Then you should consider AWS Fargate with ECS. This would launch your on-demand Tasks as individual compute and you wouldn't need to manage EC2 capacity.
- ECS also just launched new feature called scale-in protection. So you could tag the ECS Task running your algorythm as protected. This would prevent the capacity provider from scaling that specific EC2 instance down. You would have to implement this protection and removal of protection as part of your web service.
