ECS: Tasks deployed on a terminating EC2 instance
Hello,
I have an web application which runs on ECS, with autoscaling enabled on service level. The deployment are automated with CodeBuild with bluegreen configuration. Recently, we encountered error on the webapp which are caused by a bad assignation from ECS to EC2 instance which are in terminating status. Theses EC2 instances were being terminated by the autoscaling.
Is there a way to prevent ECS to assign new containers on terminating EC2 instances ?
- Più recenti
- Maggior numero di voti
- Maggior numero di commenti
yes there is a way. ECS provides a features called Container Instance Draining" which allows you to gracefully remove instances from your ECS cluster without disrupting tasks running on those instances.
Here's how you can enable container instance draining in ECS: Enable Container Instance Draining: In the ECS console, navigate to the "Clusters" page -> Select your cluster -> Click on the "ECS Instances" tab -> Select the instance(s) that you want to enable draining for -> Choose "Update Container Instances State" from the "Actions" dropdown menu->Select "DRAINING" from the dropdown menu and click "Update Instances State".
since you are using Auto Scaling Groups (ASGs) to manage your EC2 instances, configure the ASG to wait for instances to be drained before terminating them.Set the "Termination Policy" of the ASG to "OldestInstance" or "Default" to allow instances to be drained before termination.
Open the AWS Management Console and navigate to the EC2 Dashboard -> In the navigation pane, under "Auto Scaling", select "Auto Scaling Groups" Select the Auto Scaling Group (ASG) that manages your EC2 instances->Click on the "Details" tab, then scroll down to the "Advanced" section ->Under "Instance refresh settings", enable "Instance Refresh" and configure the desired settings for instance draining-> Ensure that the "Minimum healthy percentage" is set to an appropriate value to allow for draining
Adjust your scaling policies to ensure that new instances are launched before existing instances are terminated. This can help maintain the desired capacity of your ECS cluster during scaling events :- https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scaling-simple-step.html
Configure health checks for your ECS tasks to ensure that unhealthy tasks are stopped and replaced with healthy ones on other instances :- https://aws.amazon.com/blogs/containers/a-deep-dive-into-amazon-ecs-task-health-and-task-replacement/
Hope it clarifies and if does I would appreciate answer to be accepted so that community can benefit for clarity, thanks ;)
Contenuto pertinente
AWS UFFICIALEAggiornata 3 anni fa- AWS UFFICIALEAggiornata 2 anni fa