How to return an Instance to the warm pool via CLI?

0

Hi,

How can I return an instance to the warm pool via the CLI, similar to what is possible with aws autoscaling terminate-instance-in-auto-scaling-group --region "$1" --instance-id "$2" --should-decrement-desired-capacity? Ideally, I would like a command such as: aws autoscaling stop-instance-in-auto-scaling-group --region "$1" --instance-id "$2" --should-decrement-desired-capacity.

I currently have instance scale-in protection enabled for my instance therefore I'd like to scale-in from the instance itself using the CLI.

Thanks

Ashaman
asked 4 months ago193 views
2 Answers
0

I have a cluster of build agents that pull jobs from an (non-AWS) queue and runs the jobs. We have a feature of the build agents wherein they stop themselves whenever they have been idle for X amount of minutes. We currently utilize this capability along with enabling scale-in protection for the instances in the auto-scaling group. When agent exceeds its idle timeout threshold, it stops itself and part of its stop sequence is to utilize the aws autoscaling terminate-instance-in-auto-scaling-group --region "$1" --instance-id "$2" --should-decrement-desired-capacity to decrement the desired capacity for the ASG and terminate itself atomically.

I am investigating leveraging warm pools for the ASG to instead of terminating the instance, rather the agent should return itself to the warm pool and decrement the desired count of the ASG. As an aside, if I replace the terminate call with a stop call, will that automatically return the instance to the warm pool? That would be progress but I would still be missing the ability to decrement the desired account for the ASG atomically with the stop operation.

Hope this helps. Thanks!

Ashaman
answered 4 months ago
  • changing the API call to StopInstances wouldn't return the instance to the warm pool. Since StopInstances is an EC2 API, the ASG isn't aware of it and would just treat the instance as failing healthchecks. Do your instances have long startup times? If not, the engineering effort to add Warm Pools might not be worth it vs just launching new instances like you're doing now.

    If the Warm Pools are worth it for you, I edited my above answer with some steps you could take to accomplish this

0

Hi @Ashaman Currently, there isn't an API which directly sends a specific instance back to a warm pool. Can you provide some more details on the exact business usecase you're trying to achieve and workflow you'd be using this command in?

EDIT: Adding steps to be able to move instances to warm pool when idle

  1. Scale-in when there's idle instances. Scaling policy ideas:
    1. Looking at CPU to find idle instances; maybe a step scaling policy with an alarm set for the MIN statistic, so that a single idle instance triggers it (whereas average might not get moved much by a single idle instance)
    2. If you have metrics in CloudWatch for the non-AWS queue size, you could use metric math to scale based on messages/instance. Or just scale-in whenever queueSize=0
  2. Disable scale-in protection on an instance when its been idle for X time (don't toggle protection on/off between jobs, otherwise you increase the risk of API throttling)
  3. allow the combination of the above to naturally scale-in idle instances, and have the ASG set to scale-in to the warm pool
AWS
answered 4 months ago
profile picture
EXPERT
reviewed 24 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions