EC2 instance terminated with active processes

0

(using Elastic beanstalk with application Load balancer) I have a long running process from an endpoint ,and in the middle of ongoing process the LB scales down and terminates the instance.

AIM: to allow the instance to finish the ongoing process completely (and divert new incomming request to other instance preferabley) and after that terminate.

found this in linked autoscaling policy: (breaches the alarm threshold: NetworkOut < 2000000 for 1 consecutive periods of 300 seconds for the metric dimensions: AutoScalingGroupName = awsebxxxxxxxxxx Remove 1 capacity units 360 seconds before allowing another scaling activity)

  • There isn't enough info here. Depending on your usecase, either of the below answers might work. The Lifecycle Hook of an ASG starts after the instance is deregistered from the ELB. If the long running process is async and can finish on its own; a Hook would help. If there's a single long running request from a client waiting for the response, longer Deregistration Delay would help.

    If the client sends a new request to get the answer, you'll need to rearchitect so the result of the long running process is stored outside of the instance and any instance in the fleet can provide the answer

  • yes you are right there is not enough info here, the process can finish on its own and could take around 70 minutes in worst case. the client is not waiting for the response. the client does not sends a new request to get the answer.

  • A Lifecycle Hook (LCH) should work great here then. Add a Terminating LCH to the ASG, which will keep the instance in Terminating:Wait until the hook is completed. Add a cron to the instance regularly polling IMDS to see if the instance has started its LCH. If it has, wait for the job(s) to finish and then complete the hook. See LCH section: https://docs.aws.amazon.com/autoscaling/ec2/userguide/gracefully-handle-instance-termination.html

Shik12
已提问 9 个月前554 查看次数
2 回答
1
已接受的回答

You can set the deregistration delay on the Application Load Balancer target group. https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#deregistration-delay

The autoscaling group sends a deregistration call to the ALB once a scale-in event happens, and the ALB then waits for the configured deregistration delay. During that period, no new requests will reach the target group. After that period elapsed, the instance(s) get terminated. The downsides here are (1) that the ALB will wait even if there are no more critical processes running on your instance. (2) You would need to make the delay large enough to ensure that the processes have enough time to finish.

If you observe a scale-in event in the console, you can see the activity "WaitingForELBConnectionDraining" before the instance gets terminated.

profile pictureAWS
已回答 9 个月前
0

Hi,

the instance was terminated because of the defined scaling policy: alarm threshold: NetworkOut < 2000000 for 1 consecutive periods of 300 seconds

This states that after low network out activity for 5 minutes (300 seconds), an instance is going to be terminated.

To allow the instance to complete the long running process, you can use lifecycle hooks

Amazon EC2 Auto Scaling offers the ability to add lifecycle hooks to your Auto Scaling groups. These hooks let you create solutions that are aware of events in the Auto Scaling instance lifecycle, and then perform a custom action on instances when the corresponding lifecycle event occurs. A lifecycle hook provides a specified amount of time (one hour by default) to wait for the action to complete before the instance transitions to the next state.

profile pictureAWS
专家
已回答 9 个月前
  • if my aim is just to let the process complete, what action are you suggesting me to add in that hook?

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则