How do I troubleshoot connections that drop or intermittently fail with my Network Load Balancer?

Resolution

Change your client-server architecture

If the client and server applications run on the same Amazon Elastic Compute Cloud (Amazon EC2) instance, then connections time out. The timeout can happen when the client tries to connect through an internal Network Load Balancer with client IP preservation turned on.

To resolve the timeouts, turn off client IP preservation on the internal Network Load Balancer for TCP and TLS target groups. Or, move the client application to a different instance from the target. Connections then work as expected even if you turn on client IP preservation.

Configure security group rules for client IP preservation

If you turn on client IP preservation, then the source IP address of data traffic that reaches the targets is the client IP address. Configure security groups on the targets to allow traffic.

Note: Health check traffic is from the private IP addresses of the Network Load Balancer, regardless of the client IP preservation setting.

Note: Client IP preservation is on by default for target groups that use UDP, TCP_UDP, QUIC, or TCP_QUIC protocols. You can't turn it off. Solutions that involve client IP preservation apply only to TCP and TLS target groups. For UDP, TCP_UDP, QUIC, or TCP_QUIC target groups, use security group configurations instead.

If the Network Load Balancer has an associated security group, then add a rule to the target security group. In the rule, reference the security group of the Network Load Balancer as the source. This security group reference allows both data traffic and health check traffic when you turn on client IP preservation.

Note: It's a best practice to associate a security group with a Network Load Balancer at the time that you create it.

If the Network Load Balancer doesn't have an associated security group, then you can't add a security group later. Configure the target security groups to keep or turn off client IP preservation.

To keep client IP preservation when clients use a small range of IPs, allow traffic from client IP addresses on the traffic port. Also, allow traffic from the private IP addresses of the Network Load Balancer on the health check port.

Note: Security groups limit the number of rules, so client IP preservation becomes difficult when there are many client IPs to allow.

To turn off client IP preservation, turn off client IP preservation on the target group attributes. Then, allow traffic from the private IP addresses of the Network Load Balancer on the target security group. For broader coverage, specify the subnet CIDR or Amazon Virtual Private Cloud (Amazon VPC) CIDR of the Network Load Balancer.

Note: If you turn off client IP preservation, then you can still retrieve the original client IP with Proxy Protocol v2 headers.

Resolve intermittent connection failures from 5-tuple collisions

If you turned on both the client IP preservation and cross-zone load balancing, then 5-tuple collisions cause intermittent connection failures.

Check for collisions if you observe the following:

• Intermittent connection failures from clients behind a NAT device.
• The number of registered targets is small.
• Both client IP preservation and cross-zone load balancing are on.

Check the TCP_Target_Reset_Count Amazon CloudWatch metric for increases that correlate with the connection failures. The target sends TCP RST packets when it receives a duplicated 5-tuple. To confirm the issue, query Amazon VPC Flow Logs for identical 5-tuples with conflicting statuses. Look for connections from the same source IP and port that reach different Network Load Balancer node IP addresses.

To resolve issues with 5-tuple collisions, take the following actions:

• Increase the number of targets in the target group to reduce the probability of two connections being routed to the same target.
• Turn off client IP preservation on the target group so the Network Load Balancer uses its own IP as the source, eliminating the 5-tuple collision. This option is available only for TCP and TLS target groups.
• Turn off cross-zone load balancing to prevent connections through Network Load Balancer nodes in different Availability Zones from being routed to the same target.
• Expand the source port range on the client or NAT device to reduce the chance of reusing the same source port. On Linux clients, increase the ephemeral port range using the net.ipv4.ip_local_port_range kernel parameter.

Fix cross-zone load balancing issues

Intermittent connectivity issues can occur when the Network Load Balancer is deployed across multiple Availability Zones but targets are registered in only some zones. When you turn off cross-zone load balancing, only the Network Load Balancer node in the matching Availability Zone forwards traffic to targets. If Network Load Balancer nodes receive requests in Availability Zones without registered targets, then the request fails and causes intermittent connection problems.

To resolve this issue, turn on cross-zone load balancing for your Network Load Balancer.

Note: If you turn on cross-zone load balancing, then you incur Amazon EC2 data transfer charges for traffic that crosses Availability Zones.

Fix idle timeout and keepalive issues

If the client and target don't send data through the connection for longer than the idle timeout, then the Network Load Balancer closes the connection.

To resolve timeout issues, take the following actions:

• Configure TCP keepalives on either the server or client side. Then, set the keepalive interval shorter than the configured idle timeout of the Network Load Balancer. For details on how to configure TCP keepalives at the OS level and in application code, see Implementing long-running TCP Connections within VPC networking.
• If you can't update client behavior, then adjust the idle timeout setting for the Network Load Balancer listener ports. For more information, see Connection idle timeout.

Resolve ephemeral port allocation errors

If you turn off client IP preservation or traffic arrives through AWS PrivateLink, then the Network Load Balancer supports approximately 55,000 simultaneous connections per target. If you exceed the port allocation, then new connections fail.

To monitor the connections, review the PortAllocationErrorCount CloudWatch metric.

To distribute connections across more targets and reduce the per-target connection count, add more targets to the target group. Then, add secondary IP addresses to the Network Load Balancer, up to 7 per subnet, to increase the available connections. Each additional IP address increases the simultaneous connections available per target.

Related information

How do I identify the source of connection resets with my Network Load Balancer?

How do I troubleshoot connections that fail to establish with my Network Load Balancer?

Troubleshoot your Network Load Balancer

Target groups for your Network Load Balancers