AWS batch does not scale down EC2 instances

Question

We have an AWS Batch compute environment that is set up to use EC2 On-demand instances.

Whenever we run a large number of tasks, Batch scales our group of instances up to the number of CPUs we require, but it never ends up scaling them back down, even when we have 0 Batch tasks running.

Looking at the autoscaling group that Batch manages, I see it has scale-in protection enabled. This means it doesn't terminate EC2 instances when the desired capacity is decreased.

So - looking at the autoscaling group's logs, it always scales up correctly, but scaling down fails because of the scale-in protection. For example:

```
Status: CANCELLED
Error: Could not scale to desired capacity because all remaining instances are protected from scale-in.
Cause: At 2022-02-02T13:11:53Z a user request update of AutoScalingGroup constraints to min: 0, max: 18, desired: 18 changing the desired capacity from 96 to provide the desired capacity of 18. At 2022-02-02T13:12:05Z group reached equilibrium.
```

My guess is this is why it isn't scaling down. But since this auto scaling group is managed by Batch and Batch should scale instances down relatively quickly, it doesn't make sense to me.

This is costing us a lot in unused EC2 time, so if anyone can offer any insights, that'd be great.

Answer

Hey Jacques, It looks like your auto-scaling group is configured to protect instances from scale-in operations.

To change the instance scale-in protection setting for your auto-scaling group:

1) Open the Amazon EC2 Auto Scaling console at https://console.aws.amazon.com/ec2autoscaling/.

2) Select check box next to the Auto Scaling group.

3) A split pane opens up in the bottom part of the Auto Scaling groups page, showing information about the group that's selected.

4) On the Details tab, choose Advanced configurations, Edit.

5) For Instance scale-in protection, de-select "Enable instance scale-in protection".

6) Choose Update.

For more info, please visit [Modify the instance scale-in protection setting for a group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-instance-protection.html#instance-protection-modify).

Note: When the instance scale-in protection setting is enabled, all current instances launched after enabling it will have instance scale-in protection enabled. You might have to disable termination protection on each instance.

To change the instance scale-in protection setting for the current instances:

1) Open the Amazon EC2 Auto Scaling console at https://console.aws.amazon.com/ec2autoscaling/.

2) Select the check box next to your Auto Scaling group.

3) A split pane opens up in the bottom part of the Auto Scaling groups page, showing information about the group that's selected.

4) On the Instance management tab, in Instances, select all of your instances.

5) Choose Actions, Remove scale-in protection. When prompted, choose Remove scale-in protection.

For more info, please visit [Modify the instance scale-in protection setting for an instance](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-instance-protection.html#instance-protection-instance).

Please let me know if this works or if you have any further concerns and I will be happy to help!

AWS batch does not scale down EC2 instances

Relevant content