Healthcheck latency between AWS Auto Scaling and Application Load Balancer

0

** Background**

Hello. I spent 2 days solving my problem, reading AWS docs and articles on different sites without any luck. I have EC2 instance which is connected to Application Load Balancer(ALB) with HTTP listener, and I use Autoscaling Group(ASG) for this ALB. I'm trying to set the lowest latency as possible between Target Group(TG) healthcheck status "unhealthy" and new ready running instance, running by ASG. Instance start up from "Golden Copy" AMI and it come ready in 60 seconds. My settings for TG are next:

Health Check Interval Seconds: 10
Health Check Timeout Seconds: 6
Healthy Threshold Count: 3
Unhealthy Threshold Count: 3
Deregistration Delay: 0

So, I connect with ssh to instance, make some changes in web server config to forcefully respond with 404 status for health check. Health status on TG changes to "unhealthy" in approximately 15-20 seconds.

Problem

The problem is that it takes 80-90 seconds for Health status to be changed to "unhealthy" in ASG, on Instance management page. And ASG triggers instance replacement only in 80-90 seconds, which is too much.

Question

How to decrease latency between TG health status and ASG health status?

1 Answer
1
Accepted Answer

There are different mechanisms for each service that are asynchronous from each other. Because they run on different

ALB

  • You set the interval and consecutive unhealthy values. Together these determine how long it will take for an instance to show as unhealthy on the ALB itself. As soon as its marked unhealthy here, the ALB will stop sending new requests to the instance (unless all targets in that target group are unhealthy, and then it will fail open)

ASG

  • AutoScaling has a separate internal interval where it checks to see if an instance is unhealthy or not by looking at the instances status in EC2 and on any load balancers (if ELB healthchecks are enabled on the ASG)
  • The next time the ASG healthcheck happens after your ALB marks the instance unhealthy, the ASG will then mark it unhealthy. The ASG healthcheck interval is not configurable
  • The next ASG launch/terminate cycle will then lead to the instance being replaced

Amazon EC2 Auto Scaling checks that all instances within the Auto Scaling group are running and in good shape by periodically checking the health state of the instances.(1)

(1) https://docs.aws.amazon.com/autoscaling/ec2/userguide/healthcheck.html#determine-instance-health


I've reached out to our internal docs team asking to make this line in the ELB section (right below the above mentioned line) a bit more explicit about these 2 asynchronous timers. If there's any other sections/docs you feel could be improved around this topic you can either comment here and I'll pass it on; or click the 'feedback' button on a given doc, which sends your feedback directly to the document team that works on that service.

you can configure the group to mark an instance as unhealthy when Elastic Load Balancing reports it as unhealthy

AWS
answered 2 years ago
  • Thank you for your help. I appreciate it.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions