ASG suspend Terminate does not work during ASG rolling update triggered by cloud formation

0

Hi there, I'm trying to control the speed how fast ASG is replacing the instances, I'm trying to suspend terminate process during a ASG rolling update triggered by cloud formation, but this does not prevent more instances being terminated and I can configure the ASG does show Terminate as suspended processes. Anything I missed here or is there any other way I can try slow down how fast ASG is replacing the instances?

asked 22 days ago37 views
2 Answers
0

It appears that suspending the Terminate process in an Auto Scaling Group (ASG) during a rolling update triggered by CloudFormation may not have the desired effect of slowing down instance replacement. This is because the Terminate process suspension primarily affects regular scaling activities and not necessarily the instance replacement process during updates.

To address your goal of slowing down how fast the ASG replaces instances during a CloudFormation-triggered update, you might consider the following approaches:

  1. Use CloudFormation update policies: Instead of relying on ASG process suspension, you can use CloudFormation's UpdatePolicy attribute to control the instance replacement process. This allows you to specify parameters like MinInstancesInService, PauseTime, and WaitOnResourceSignals to manage the pace of updates.

  2. Implement Lifecycle Hooks: You can set up lifecycle hooks for the Terminating:Wait event in your ASG. This will pause the termination process of an instance, giving you more control over when instances are actually terminated. You can then use a custom script or Lambda function to manage the timing of instance terminations.

  3. Adjust CloudFormation stack update options: When updating your CloudFormation stack, you can use options like --max-batch-size and --min-successful-instances-percent to control the number of instances updated in each batch and the success criteria for moving to the next batch.

  4. Use a custom resource in CloudFormation: Create a custom resource that manages the instance replacement process according to your specific requirements. This gives you fine-grained control over the timing and conditions of instance replacements.

Remember that while suspending the Terminate process can affect regular scaling activities, it may not directly impact the instance replacement process during a CloudFormation-triggered update. The above alternatives provide more direct control over the update process and can help you achieve the desired slower replacement rate.
Sources
Gracefully scale down AWS ASG | AWS re:Post
schedule to update all autoscaling groups in my account using cloudformation | AWS re:Post

profile picture
answered 22 days ago
  • I already have update policy and lifecycle in place, what I want to do is to not have too many instances in terminating state (due to there is work required in termination lifecycle) at one time but also don't want to block launching new instances to serve the traffic.

0

CloudFormation (CFN) Rolling Updates terminate instances via the TerminateInstanceInAutoScalingGroup API call. This API bypasses the Terminate suspended process, which is why you see terminations continue

Rolling Updates don't generally care about how long it takes to Terminate instances, just that the new replacement instances are InService and there is a cfn-signal received. You could try either delaying sending the cfn-signal on the new instance, or delaying completing the Lifecycle hook.

Setting BatchSize to 1 on the UpdatePolicy would also slow down the process, but that's still not directly looking at how long terminations take, just a blunt force tool for slowing the process down

AWS
EXPERT
answered 18 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions