What is the ASG's typical behaviour if instance launch fails from warmpool for first time?

Question

Context:
Our applications have very high cold start (around ~5 mins), so we have warmpool enabled in our ASGs. However we have been observing multiple cases where launching an instance from warmpool fails due to `Insufficient Capacity in AZ`. In such cases, we have observed (from ASG activity) that once ASG receives instance launch failure from warmpool, it goes ahead to launch a fresh new instance altogether and moves existing warmpool instance to termination state.

This leads to high cold starts for subsequent requests and a warmpool instance is wasted for no reason.

Question: 
- Does ASG retry launching an instance from warmpool again from different AZ in such cases or does it directly launch a fresh new instance?
- Is there a way to force ASG to always launch a new instance from warmpool only?

Accepted Answer

When an instance is stopped in a Warm Pool, there isn't a capacity guarantee.  Just like starting any other stopped EC2 instance, there could be a lack of capacity for that instance type in the given AZ.  When any launch failure happens for a warm pool instance being started, including ICE (Insufficient Capacity Errors), the instance will be considered a launch failure and terminated.

>    Does ASG retry launching an instance from warmpool again from different AZ in such cases or does it directly launch a fresh new instance?

Not at this time, however this is something which could be added in the future.  I've passed your feedback on to the AutoScaling team for their awareness

---
>    Is there a way to force ASG to always launch a new instance from warmpool only?

No.  If there is no capacity in the warm pool, the ASG will launch a cold start

Answer

Hi,

You case is clearly mentioned in Limitations of https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-warm-pools.html#warm-pools-limitations

```
If your warm pool is depleted when there is a scale-out event, instances will launch 
directly into the Auto Scaling group (a cold start). You could also experience cold starts 
if an Availability Zone is out of capacity.
```

So, cold starts are normal behaviour when AZ is out of capacity.

To try to go around your issues, your may want to activate predictive autoscaling (if not done yet): https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-predictive-scaling.html

In that case, your instance(s) will be started before they are really needed and then you may hide (part of) the long start time of your application.

Best,
Didier

What is the ASG's typical behaviour if instance launch fails from warmpool for first time?

Relevant content