Network Load Balancer Stickiness Timeout with Unhealthy Nodes

How long will a flow remain to an unhealthy target? **tl;dr **Either forever or it will be immediately failed, depending on your setting and the listener/traffic protocol TCP or UDP.

• For TCP, it depend if you have flow termination to unhealthy targets on or off (default off). If on, the flow will continue, if off any packets from the client will receive TCP resets as long as the target is healthy.
• For UDP, as soon as the target is unhealthy, the next packet will get routed to a new target and that flow will persist until broken/that target goes unhealthy.

The underlying reason is that these are 2 independent behaviors:

1. Existing flows: The routing of traffic on an established flow to an unhealthy target. For TCP, an established flow behavior does not relate to the stickiness setting. This is controlled by the Target Group setting for 'target_health_state.unhealthy.connection_termination.enabled'. If it is ON, a client sending data on an established flow will receive TCP resets from the NLB, and the NLB will emit 'TCP_ELB_Reset_Count' metrics, 1 for each reset. If the setting is OFF [default], an established flow to a target that transitions from healthy to unhealthy will not be affected (allowed to continue).
2. New flows: NLB stickiness is implemented by reducing the client tuples used in our routing decision - the routing decision mechanism isn't changed, we are still creating a hash map of available targets and then flow hashing the source and picking the closest member target. Health affects this because the hash map only includes healthy targets, so if a target becomes unhealthy, the entire hash map changes. The same is true if you add or remove targets, the hash map changes. The hash map is used whenever a new flow is received. This means that a target becoming unhealthy will change the hash map and cause ALL new flows to choose new targets, which are not guaranteed to be the same target. The same thing happens after anything changes the set of if you register or de-register targets, or turn on or off cross zone load balancing.

To be explicitly clear, anything that changes the list of available targets, causes stickiness to change. This applies to ALL new flows, for all clients.

When you have a Network Load Balancer (NLB) with stickiness enabled using source_ip, the behavior regarding unhealthy nodes is designed to maintain high availability while attempting to preserve session affinity. Here's how it works:

1. If a target becomes unhealthy, the NLB will stop routing new connections to that target immediately.

2. For existing connections that were "stuck" to the unhealthy target due to source IP stickiness, the NLB will continue to route traffic to the unhealthy target for a short period. This is because the NLB operates at the transport layer (Layer 4) and doesn't have visibility into the application-level health of existing connections.

3. The exact duration for which traffic continues to be sent to an unhealthy target is not explicitly defined and can vary. It's generally a short period, typically on the order of seconds to a few minutes.

4. After this short period, or when the unhealthy target is deregistered from the target group, the NLB will route subsequent traffic from that source IP to a healthy target.

It's important to note that there isn't a configurable "stickiness timeout" for NLBs like there is for Application Load Balancers. The source IP stickiness for NLBs is designed to be long-lived but will be overridden when targets become unhealthy or are deregistered.

Regarding your question about checking when the switchover occurred for a given source IP, unfortunately, the NLB doesn't provide built-in logging or metrics at that level of granularity. To track such switchovers, you would need to implement application-level logging or monitoring on your targets.

For UDP traffic, which you mentioned you're using, the behavior is similar. However, it's worth noting that UDP is a connectionless protocol, so the concept of "existing connections" doesn't apply in the same way as it does for TCP. The NLB will still attempt to maintain stickiness based on the source IP, but the stateless nature of UDP means that each packet is treated independently.

Sources
Troubleshoot stickiness issues on your Network Load Balancer | AWS re:Post
TargetGroupAttribute - Elastic Load Balancing