unhealthy host in network load balancer

Can you clarify if you are using AWS API Gateway and AWS Network Load Balancer. I do not think the AWS API Gateway and NLB has direct integration option.

Assuming third party API Gateway and AWS NLB, have you configured health checks for API Gateway. What happens when it's unhealthy, does it still respond with status code 200. If so, then NLB may not be able to identify the unhealthy instance.

Also, check your unhealthy threshold in the target group setting and the interval. For example, if you set unhealthy threshold = 10 and interval = 300 seconds, it can take 50 minutes before it is considered unhealthy.

Btw, cross zone load balancing is irrelevant for this question.

The behavior you're observing with your Network Load Balancer (NLB) continuing to direct traffic to an unhealthy target is related to how NLBs handle existing connections and new flows when a target becomes unhealthy. Here's what's likely happening:

1. For existing connections: By default, NLBs have a feature called "connection termination" enabled. When this is on, the load balancer will terminate established connections to targets that become unhealthy. However, if this setting is disabled, existing connections to an unhealthy target will be kept active, allowing them to close gracefully. This could explain why you're seeing traffic still being directed to the unhealthy target in your flow logs.

2. For new connections: When a target becomes unhealthy, the NLB should immediately stop routing new connections to that target. Instead, new connections should be routed to healthy targets, even across zones if cross-zone load balancing is enabled.

3. Stickiness: If you have source IP stickiness enabled on your NLB, this could also impact routing behavior. When stickiness is enabled, the NLB attempts to route traffic from a specific source IP to the same target, even if that target becomes unhealthy, for a short period.

4. Fail-open mode: In a scenario where all targets become unhealthy, the NLB enters a "fail-open" mode. In this state, it will route requests to all registered targets, regardless of their health status. This is done to maintain some level of service rather than completely stopping all traffic.

To address this issue:

1. Check your target group attributes, specifically the "connection termination" setting. You may want to enable this if it's currently disabled.

2. Verify that your health check settings are correctly configured to accurately detect unhealthy targets.

3. If you're using source IP stickiness, consider its impact on your traffic distribution.

4. Monitor your NLB metrics, particularly the TCP_ELB_Reset_Count, which can indicate issues with unhealthy targets.

5. Ensure that your application can handle new "in-transit" flows without sending resets, which could cause client connections to restart.

Remember, the exact behavior can depend on various factors, including your specific configuration and the nature of your traffic (TCP vs UDP). If the issue persists, you may need to engage AWS support for a more detailed investigation of your specific setup.
Sources
Network Load Balancer Stickiness Timeout with Unhealthy Nodes | AWS re:Post
Edit target group attributes for your Network Load Balancer - Elastic Load Balancing
Troubleshoot your Network Load Balancer - Elastic Load Balancing