How do I configure my capacity provider to retain Amazon EC2 instances in an Amazon EC2 Auto Scaling group that has running Amazon ECS tasks?

Short description

If you use a capacity provider, then configure your Amazon ECS cluster settings to avoid disrupting running tasks when the cluster scales down. Implement scale-in instance protection in the EC2 Auto Scaling group and managed scaling protection in the Amazon ECS capacity provider settings. This configuration allows Amazon ECS to retain instances with running tasks.

Or, configure Amazon ECS to gracefully drain tasks when you stop instances, and protect specific instances from premature termination.

Your configuration uses the following components:

• Capacity providers: Capacity providers use Auto Scaling groups to dynamically provision EC2 instances.
• Instance draining: When an EC2 instance in an Auto Scaling group is marked for termination, it enters a DRAINING status and no longer accepts new tasks. Running tasks must either finish or move to another instance.
• Cluster auto scaling: Cluster auto scaling uses capacity providers to automatically scale Auto Scaling groups based on task requirements.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Activate scale-in protection for your Auto Scaling group

To use managed termination protection in cluster auto scaling, activate instance scale-in protection on the Auto Scaling group. This allows the Auto Scaling group to retain instances with running tasks.

To activate scale-in protection, run the following update-auto-scaling-group AWS CLI command:

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name my-asg \
  --new-instances-protected-from-scale-in

Note: Replace my-asg with your Auto Scaling group name.

Then, run the following put-cluster-capacity-providers command to activate managed termination protection so that Amazon ECS retains instances with running tasks:

aws ecs put-cluster-capacity-providers \
  --cluster my-cluster \
  --capacity-providers my-capacity-provider \
  --enable-managed-scaling \
  --managed-termination-protection ENABLED

Note: Replace my-cluster with your cluster name and my-capacity-provider with your capacity provider name.

Configure Amazon ECS to gracefully drain tasks when you remove an instance

Gracefully drain tasks when you remove them from an instance. Use managed instance draining to allow Amazon ECS to reschedule your tasks on other instances before termination.

To activate managed instance draining, run the following update-capacity-provider AWS CLI command:

aws ecs update-capacity-provider \
--name my-cp \
--auto-scaling-group-provider '{
  "managedDraining": "ENABLED"
}

Note: Replace my-cp with your capacity provider name.

Amazon ECS allows a maximum of 48 hours to drain instances. If tasks remain on the instance after this period, then Amazon ECS terminates the instance and stops the remaining tasks.

Retain a specific critical task

Use task protection to prevent Amazon ECS scaling or deployment actions from stopping critical tasks.

Important: Even when you activate task protection, Amazon ECS can still terminate the underlying instance. Task protection only prevents Amazon ECS from stopping tasks.

To activate task protection, run the following update-task-protection AWS CLI command:

aws ecs update-task-protection \
--cluster my-cluster \
--tasks my-task-id \
--protection-enabled

Note: Replace my-cluster with your cluster name and my-task-id with your task ID. By default, protection lasts for 2 hours. To change this duration, use the --expires-in-minutes option.

To check the status of task protection in your cluster, run the following get-task-protection AWS CLI command:

aws ecs get-task-protection \
--cluster my-cluster \
--tasks my-task-id

Note: Replace my-cluster with your cluster name and my-task-id with your task ID.

Troubleshoot further issues

"If you activate protection for your Auto Scaling group, capacity provider, and tasks, and still encounter issues, then complete the following actions:

• Check the Amazon ECS agent logs for task draining issues.
• Make sure that the Amazon ECS agent is running and correctly configured on your instances.
• Verify that your AWS Identity and Access Management (IAM) task roles have the permissions that the containers in your tasks must assume.

Related information

Protect your Amazon ECS tasks from being terminated by scale-in events

GetTaskProtection

UpdateTaskProtection

Elastic Container Service (ECS) Task Protection Examples on the GitHub website