Empty Target Group Keep Receiving Traffic

Question

I've the following architecture:

1. Weight based elastic load-balancer [20% on my main target group #2.1, 80% on my spot target group #2.2]
2. Two Target groups:
    1. Main Target Group: which contains on-demand EC2 instances [currently contains 1 instance only]
    2. Spots Target Group: which contains initiated EC2 instances by spot request [spot request with termination policy when an instance exceed predefined cost limit, the instance gets terminated and lost]
3. Route53 that route the traffic to ELB by Alias [#1]

Recently my spot EC2 instance gets terminated which result spot target group to be empty, after that I started to receive 503 http response code when I sent a request over the load-balancer, and that happens only when the traffic gets routed by ELB to the spot target group.

Based on this [article ](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health.html), empty target groups is considered as unhealthy, so why in my case the empty target group keep receiving traffic while it's unhealthy? is that the expected flow? what are the possible solutions to avoid that case?

Thanks.

Behrang_KS · Answer

If a target group contains only unhealthy registered targets, the load balancer routes requests to all those targets, regardless of their health status. This means that if all targets fail health checks at the same time in all enabled Availability Zones, the load balancer fails open. The effect of the fail-open is to allow traffic to all targets in all enabled Availability Zones, regardless of their health status, based on the load balancing algorithm : https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html

To avoid this scenario you might want to deploy a different logic of LB using a non-spot baseline : https://aws.amazon.com/blogs/compute/running-web-applications-on-amazon-ec2-spot-instances/

jwesley · Answer

That IS the expected flow. Your weighted routing policy is still sending 80% of your traffic to the spot target group, even when empty. The simplest fix would probably be to create a new launch template that puts your spot instances in an auto-scaling-group. You can use a mixed-instance policy in the ASG to make sure not all of your spot fleet is stopped at once.

Empty Target Group Keep Receiving Traffic

Relevant content