How to scale in ec2 fargate containers if job is done?
We want to run containers on an ec2 fargate cluster that will run jobs of an arbitrary duration (30 minutes on average). We want to scale out based on active TCP connection metrics via the AWS loadbalancer (1 tcp connection contains stream that needs to be handled by 1 container). This seems possible with cloudwatch metrics. The question is how to scale in once the job is done. We do not want to stop a container when it's still processing an active connection. It could take hours for the job to finish (and that is OK).
Is there a way we can remove a container once it is done processing (aka no active tcp connection anymore?) If the process inside the container stops, is it removed from the desired active container total?
If you need to trigger a Fargate task that executes a batch job and then exits wouldn't make more sense to use a pattern such as the one I am describing in this blog post? Note that in my example I submit an AWS Batch job but API GW also supports launching an ECS task (on Fargate).
The idea is that the task would launch a program that does what you need it to do and then exits (you should not worry about scaling in at all with this pattern). Need another task? Hit the API GW API to submit/launch another job/task and that's it.
Generally speaking, systems on AWS are conservative about scale ins, compared to scale out.
You do not want your system to be scaled in immediately after it completes certain tasks.
That being said, from a long term auto scaling policies such as target tracking do a fairly good job of matching capacity with demands. If there is no connection to a container, the auto scaling scale in event should be triggered.
Also, to another point in the question, you might want to check the de-registration delay value for your ELB as well to avoid terminating active long-running connections.
Relevant questions
Moving to ECS-Fargate from EC2
asked 4 months agoEC2 - Windows 2019 with containers: call service on host
asked 3 years agoUse API Gateway to run a Fargate task and then terminate that pod upon completion
asked 2 months agoHow to scale an aws ecs service based on multiple alarms
asked 2 months agoHow to scale in ec2 fargate containers if job is done?
asked 2 months agoFargate: how to stop task after finishing
Accepted Answerasked 4 months agoIs it possible to use Lambda functions along with other services to scale up and scale down(probably to 0 instances) Ec2 Deployed apps
asked 2 months agoCan i access to Fargate windows containers?
Accepted Answerasked 3 months agoARM64 support for Batch jobs on AWS Fargate
Accepted Answerasked 3 months agoConverting a AWS Fargate instance to AWS EC2
asked 2 years ago