Sequence of events when an asg receives a capacity rebalance recommendation or spot termination notification

When an Auto Scaling group (ASG) with mixed instances running 100% Spot receives a capacity rebalance recommendation or Spot termination notification, and there is insufficient Spot capacity in the specified Spot pools, the sequence of events is as follows:

1. The ASG receives the rebalance recommendation or Spot termination notification.

2. The ASG attempts to launch a new Spot Instance in other enabled Availability Zones.

3. If there is no capacity available for the specified instance types in one Availability Zone, the ASG continues trying to launch Spot Instances in other enabled Availability Zones until it succeeds.

4. The ASG does not immediately create an INSTANCE_LAUNCH_ERROR event. Instead, it keeps attempting to launch new instances.

5. While the ASG is trying to launch new instances, the previous instances that received the rebalance recommendation or termination notice will eventually be interrupted and forcibly terminated with a two-minute interruption notice.

6. In the worst-case scenario, if new instances consistently fail to launch or their health checks fail, the ASG will continue its attempts to relaunch them.

It's important to note that the ASG does not immediately give up or create an error event when it encounters capacity issues. Instead, it persists in trying to launch new instances across different Availability Zones. The ASG will continue this process until it successfully launches new instances or until the old instances are forcibly terminated.

To better manage this situation, it's recommended to:

1. Configure your ASG to use multiple instance types and Availability Zones to increase the chances of finding available capacity.

2. Implement Capacity Rebalancing, which proactively attempts to replace Spot Instances that receive a rebalance recommendation before they are interrupted.

3. Use lifecycle hooks and develop custom actions to gracefully handle instance replacements.

4. Monitor rebalance recommendations and interruption notices using Amazon EventBridge to trigger checkpoints for your workload or handle interruptions gracefully.

By following these practices, you can improve the resilience of your ASG when dealing with Spot Instance capacity constraints.
Sources
Capacity Rebalancing in Auto Scaling to replace at-risk Spot Instances - Amazon EC2 Auto Scaling
Best practices for Amazon EC2 Spot - Amazon Elastic Compute Cloud
Instance maintenance policy for Auto Scaling group - Amazon EC2 Auto Scaling

Unfortunately, there's no built-in configuration in EC2 Auto Scaling that forces the ASG to emit an INSTANCE_LAUNCH_ERROR event immediately when Spot capacity is unavailable. By design, the ASG keeps retrying silently across Spot pools and Availability Zones to find capacity. An INSTANCE_LAUNCH_ERROR is only emitted after repeated failures—and that delay varies depending on the retry backoff logic and instance launch strategy.

🔧 Workarounds & Recommendations: EventBridge Rule for Launch Failures You can create an EventBridge rule that listens for EC2 Spot Instance Interruption Warning and EC2 Instance-launch Lifecycle Action Heartbeat Timeout. While indirect, these are signals that a rebalance is underway and replacement capacity isn't being acquired fast enough.

Use Instance Refresh with On-Demand fallback You could combine Capacity Rebalancing with Instance Refresh and have a custom Lambda function or automation tool monitor ASG activity logs and proactively adjust the OnDemandPercentageAboveBaseCapacity when persistent Spot launch failures occur.

Monitor ASG Metrics via CloudWatch Track GroupDesiredCapacity and GroupInServiceInstances. If they diverge significantly and stay that way for several minutes, it's a good indicator that launch failures are occurring. This can trigger an alarm to shift traffic to on-demand instances.

Manually trigger fallback logic When you receive a rebalance recommendation or termination notice via EventBridge (or EC2 Spot Instance Interruption Warning), proactively update the ASG configuration to temporarily allow more On-Demand capacity.

TL;DR: The INSTANCE_LAUNCH_ERROR is not immediate, and currently, the only way to act right away is to monitor Spot Interruption or Rebalance Recommendation events and implement fallback logic yourself via automation (Lambda, EventBridge, etc.).

Let me know if you'd like help setting up an EventBridge pattern or automation script to handle this!