I see that associated EC2 instance has "Protected from: scale in". Why is it like this and is this what's causing EC2 instances with single daemon tasks to not be terminated?
Yes, the instance being protected from scale-in makes it so it won't be terminated (except if an issue is detected, for example a healthcheck fails, or an explicit termination request is sent for that instance)
As for why, there could be multiple reasons, a few common ones might be
- Something else (other than ECS) re-enabled it. Check Cloudtrail for SetInstanceProtection calls, and look if any aren't made by ECS.
- ECS tried to disable protection, but there was API throttling. This can again be seen in CloudTrail by searching for SetInstanceProtection calls and seeing if any of them show a RateExceeded error. If this is happening, check if you have any scripts making large amounts of AutoScaling API calls that can be reduced. If not, then open a case with support to evaluate increasing the API limit
- Verify that at the group level, the ASG (AutoScaling Group) has scale-in protection enabled, so that new instances have it on by default (ECS requires this to be enabled when you enabled Managed Termination Protection, but it may have later been toggled off)
Additionally, its possible for instances to still not get terminated when protection is removed. For an instance to be terminated, the Desired Capacity of the ASG has to go down (generally through a scaling policy lowering it). If the desired hasn't gone down, then instances won't be scaled in, even if the capacity provider has removed scale-in protection
ECS Capacity Provider Scale In Timeasked 3 months ago
Should ECS/EC2 ASGProvider Capacity Provider be able to scale-up from zero, 0->1Accepted Answer
How to add launch type to an ecs service from ecs-cli?asked 2 months ago
ECS cluster with EC2 capacity provider is left with instances having only daemon task runningasked 22 days ago
Should ECS Service Task start be triggered by ASG capacity 0->1?Accepted Answer
ECS Capacity Provider Auto-Scaler Instance SelectionAccepted Answer
AWS ECS CapacityProvider not running tasksAccepted Answer
ECS: Capacity Provider vs Autoscaling Groupasked a year ago
Running multiple container using single EC2 instances in ECSasked 4 months ago