GWLB passing traffic to "Unhealthy" Firewall

Question

Hi, I have an issue with a GWLB. The GLWB is in front of a pair of Geneve aware firewalls that are in different availability zones. I have a target group that is health checking the firewalls, the health checks pass and both firewalls go into a “healthy” state.

All works as expected, packets flow across both firewalls following AWS’ load balancing hash. I see active traffic on both firewalls. I have checked the FWs ability to map the VPCE of the GWLB and I can see these register, I have two registrations, one per availably zone.

I also have rebalancing, cross zone functionality enabled.

During testing if I purposely cause the health check to fail, for example I block the health check traffic, I can see one of the firewalls in the target group go 
into a “unhealthy” state (expected), **I wait 10 mins**.

As I’ve only impacted the health check, the firewall can still actually pass traffic, it shouldn’t as the session would be drained and the GWLB should now only send traffic to the healthy firewall, however when I packet capture on the appliance I can see new sessions (tcp/syn) being received on the “unhealthy” firewall still.

I would expect this if both firewalls failed and the GWLB went into a fail open state.

After 10mins all session should be drained, I’ve also made sure a new TCP session on different ports and sequence numbers are sent to rule out any caching issues.

You might be asking why I am testing like this, because in a true failure I noticed packet loss and it was exactly the same, packets being delivered to the “unhealthy” host..

What have I got wrong or mis-understood?

re:Post Agent · Answer

When a Network Load Balancer (NLB) or Gateway Load Balancer (GWLB) has targets that become unhealthy, there are specific behaviors that govern how traffic is handled.

In your case, there are a few key points to understand:

1. **Fail-open mode**: By default, when all targets become unhealthy, the load balancer enters "fail-open" mode, where it routes traffic to all registered targets regardless of health status. However, this shouldn't be happening in your case since you still have one healthy firewall.

2. **Minimum healthy targets threshold**: There's a configurable setting called "unhealthy_state_routing.minimum_healthy_targets" that determines when the load balancer should enter fail-open mode. If the number or percentage of healthy targets falls below this threshold, the load balancer will route traffic to all targets, including unhealthy ones. This could explain why your unhealthy firewall is still receiving traffic.

3. **Connection termination setting**: For existing connections, the load balancer has a "connection termination" setting. If disabled (which is the default), existing connections to an unhealthy target will be kept active. However, this shouldn't affect new TCP SYN packets that you're observing.

4. **Source IP stickiness**: If you have source IP stickiness enabled, the load balancer might continue routing traffic from specific source IPs to the same target, even if it becomes unhealthy, for a short period.

To resolve this issue:

1. Check your target group attributes, specifically the "unhealthy_state_routing.minimum_healthy_targets" setting. You might need to adjust this to ensure it doesn't trigger fail-open mode with just one unhealthy firewall.

2. Verify if source IP stickiness is enabled and consider its impact on your traffic distribution.

3. Ensure your health check settings are correctly configured to accurately detect unhealthy targets.

4. Consider enabling the "connection termination" setting if you want to immediately terminate connections to targets that become unhealthy.

These adjustments should help ensure that new traffic is only routed to your healthy firewall when one becomes unhealthy.  
**Sources**  
[unhealthy host in network load balancer | AWS re:Post](https://repost.aws/questions/QUB6fe6ZlMR9iR8JpBZY8Xqw/unhealthy-host-in-network-load-balancer)  
[Target Group - Only send traffic to healthy nodes | AWS re:Post](https://repost.aws/questions/QU_qxkkmJiQ_mMJClbgOLaRw/target-group-only-send-traffic-to-healthy-nodes)  
[Network Load Balancer Stickiness Timeout with Unhealthy Nodes | AWS re:Post](https://repost.aws/questions/QUQwR1pCbWST28vXu96aQfBw/network-load-balancer-stickiness-timeout-with-unhealthy-nodes)  
[Update the target group health settings for your Network Load Balancer - Elastic Load Balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/modify-target-group-health-settings.html)

GWLB passing traffic to "Unhealthy" Firewall

Add your answer

Relevant content