ECS Managed Instances - stuck on draining when Agent lost connection

When ECS Managed Instances get stuck in a DRAINING state due to a lost agent connection, you can force deregister these instances to proceed with deletion of the capacity provider. This situation commonly occurs when network connectivity is disrupted, such as when a NAT instance is terminated before proper draining completes.

The ECS agent likely remains disconnected because it cannot communicate with the ECS service. Even after restoring the NAT instance, the agent may not automatically reconnect due to network configuration issues. The agent requires proper outbound connectivity to the ECS service endpoints.

To resolve this issue, you can force deregister the stuck container instances using the AWS CLI:

aws ecs deregister-container-instance \
--cluster your-cluster-arn \
--container-instance your-container-instance-arn \
--force

This command will forcibly remove the instance from your cluster, allowing the capacity provider deletion to proceed. The --force flag is necessary since the agent is disconnected.

For future deployments, consider enabling managed instance draining on your capacity provider:

aws ecs update-capacity-provider \
--name your-capacity-provider-name \
--auto-scaling-group-provider '{
"managedDraining": "ENABLED"
}'

This helps ensure tasks are gracefully drained when instances need to be terminated. Also, ensure your network configuration includes redundancy for critical components like NAT gateways to prevent similar connectivity issues.
Sources
Troubleshooting Amazon ECS Managed Instances - Amazon Elastic Container Service
why ECS agent got disconnected | AWS re:Post
Configure capacity provider to retain instances with running tasks | AWS re:Post