Auto scaling - Capacity Rebalance kept stopping/terminating instances

Question

We are seeing that recently there seems to be issue with 
```
Capacity rebalanceInfo
When you enable capacity rebalancing, and a rebalance notification is sent to an instance, EC2 Auto Scaling automatically attempts to replace the instance before it is interrupted.
```

When enabled, we see that our instance kept terminating and stopping, which creates unnecessary loads of our cache/DB/other infra as every new instance requires setup/init
This is NOT due to spot instance being terminated as the termination reason is always
```
At 2022-10-24T01:59:17Z an instance was taken out of service in response to an EC2 instance rebalance recommendation.
```

What we expect is when an instance is only terminated only when 
1. spot capacity termination request
2. spot lifetime exceeded

Perhaps this is an issue with recent capacity limit issue w east?
```
Launching a new EC2 instance. Status Reason: Could not launch Spot Instances. InsufficientInstanceCapacity - We currently do not have sufficient capacity in the Availability Zone you requested. Our system will be working on provisioning additional capacity.
```

Matt-B · Answer

Hi There

When you enable Capacity Rebalancing for your Auto Scaling group, Amazon EC2 Auto Scaling attempts to proactively replace the Spot Instances in your group that have received a rebalance recommendation. This provides an opportunity to rebalance your workload to new Spot Instances that aren't at an elevated risk of interruption.

There are a few [best practices](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-capacity-rebalancing.html#capacity-rebalancing-considerations) to check if you are seeing frequent rebalancing recommendations.

One of which is that it is a best practice to [use a few different instance types](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html) to allow Amazon EC2 Auto Scaling to launch another instance type in the event that there is insufficient instance capacity in your chosen Availability Zones.

Also check the [allocation strategy](https://aws.amazon.com/blogs/compute/introducing-the-capacity-optimized-allocation-strategy-for-amazon-ec2-spot-instances/) and conside using the capacity-optimized strategy

Auto scaling - Capacity Rebalance kept stopping/terminating instances

Relevant content