EC2 instance terminated with active processes


(using Elastic beanstalk with application Load balancer) I have a long running process from an endpoint ,and in the middle of ongoing process the LB scales down and terminates the instance.

AIM: to allow the instance to finish the ongoing process completely (and divert new incomming request to other instance preferabley) and after that terminate.

found this in linked autoscaling policy: (breaches the alarm threshold: NetworkOut < 2000000 for 1 consecutive periods of 300 seconds for the metric dimensions: AutoScalingGroupName = awsebxxxxxxxxxx Remove 1 capacity units 360 seconds before allowing another scaling activity)

  • There isn't enough info here. Depending on your usecase, either of the below answers might work. The Lifecycle Hook of an ASG starts after the instance is deregistered from the ELB. If the long running process is async and can finish on its own; a Hook would help. If there's a single long running request from a client waiting for the response, longer Deregistration Delay would help.

    If the client sends a new request to get the answer, you'll need to rearchitect so the result of the long running process is stored outside of the instance and any instance in the fleet can provide the answer

  • yes you are right there is not enough info here, the process can finish on its own and could take around 70 minutes in worst case. the client is not waiting for the response. the client does not sends a new request to get the answer.

  • A Lifecycle Hook (LCH) should work great here then. Add a Terminating LCH to the ASG, which will keep the instance in Terminating:Wait until the hook is completed. Add a cron to the instance regularly polling IMDS to see if the instance has started its LCH. If it has, wait for the job(s) to finish and then complete the hook. See LCH section:

2 Answers
Accepted Answer

You can set the deregistration delay on the Application Load Balancer target group.

The autoscaling group sends a deregistration call to the ALB once a scale-in event happens, and the ALB then waits for the configured deregistration delay. During that period, no new requests will reach the target group. After that period elapsed, the instance(s) get terminated. The downsides here are (1) that the ALB will wait even if there are no more critical processes running on your instance. (2) You would need to make the delay large enough to ensure that the processes have enough time to finish.

If you observe a scale-in event in the console, you can see the activity "WaitingForELBConnectionDraining" before the instance gets terminated.

profile pictureAWS
answered 10 months ago


the instance was terminated because of the defined scaling policy: alarm threshold: NetworkOut < 2000000 for 1 consecutive periods of 300 seconds

This states that after low network out activity for 5 minutes (300 seconds), an instance is going to be terminated.

To allow the instance to complete the long running process, you can use lifecycle hooks

Amazon EC2 Auto Scaling offers the ability to add lifecycle hooks to your Auto Scaling groups. These hooks let you create solutions that are aware of events in the Auto Scaling instance lifecycle, and then perform a custom action on instances when the corresponding lifecycle event occurs. A lifecycle hook provides a specified amount of time (one hour by default) to wait for the action to complete before the instance transitions to the next state.

profile pictureAWS
answered 10 months ago
  • if my aim is just to let the process complete, what action are you suggesting me to add in that hook?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions