How do I troubleshoot TCP connectivity issues on my Network Load Balancer?

Resolution

To trace packets and determine where they're getting dropped, you can use either Amazon Virtual Private Cloud (Amazon VPC) Flow Logs or packet captures.

The following is an example of a simple traffic flow from an on-premises server to an internet-facing Network Load Balancer:

Client(onprem)---->Firewall(onprem)---through internet--->NLB--->target

If you receive an error, then check the networking path and the intermediary devices between the client and the target.

Troubleshoot the client connection

Identify whether the error is a DNS resolution failure, refused connection, or a timed-out connection.

Make sure that the client can resolve the DNS to the Network Load Balancer's correct IP address. Use tools such as, dig (dig www.example.com) for Linux or nslookup (nslookup.www.example.com) for Microsoft Windows.

Use dig or nslookup to resolve the Network Load Balancer's Fully Qualified Domain Name (FQDN) that's available on your AWS Management Console. The Network Load Balancer's FQDN has the following syntax:

name-id.elb.region.amazonaws.com. For example: dig my-load-balancer-1234567890abcdef.elb.us-east-2.amazonaws.com

Make sure that the IP addresses from the dig or nslookup outputs of the custom domain DNS and the load balancer's FQDN are the same.

Check if the client packets are going to your Network Load Balancer. Use packet capturing tools, such as tcpdump, wireshark, or the client's VPC flow logs to check the packets that leave and enter the client's network interface. If the SYN packets aren't going to your Network Load Balancer, then check the client application to see what's blocking traffic. Make sure that the client's operating system (OS) and firewall allow incoming and outgoing traffic for the Network Load Balancer.

You might see REJECT in the VPC flow logs for the traffic that's going to the Network Load Balancer's IP address. Check the load balance's security group and the network access control list's (network ACL) outbound rules to determine if they're allowing traffic. You might also see REJECT in the VPC flow logs for the traffic that's coming from the Network Load Balancer's IP address. Check the client's security group and the network ACL's inbound rules to determine if they're allowing traffic.

You might see response traffic (SYN_ACK) that's coming from the load balancer to the client but no ACK returned from client to the load balancer. Check if the client's firewall is blocking traffic from the load balancer.

Troubleshoot failed connections to the Network Load Balancer

Review the Network Load Balancer's VPC flow logs to determine whether the client's traffic is reaching the load balancer. If the VPC flow logs show that the load balancer rejected traffic from the client, then check the load balancer's security group and subnet's network ACL inbound rules. Make sure that the rules allow inbound traffic from the client IP address on the load balancer's listener port.

If the Network Load Balancer accepted traffic from the client, then check if the load balancer is forwarding the traffic to the target IP address. Packets that are going to the Network Load Balancer's target must have the source IP address as the load balancer's private IP address. They must also have the destination IP address as the target's IP address.

To resolve a REJECT error, check the security group and the subnet's network ACL outbound rules. Make sure that the rules allow outbound traffic to the target IP address on the target port.

Check the Network Load Balancer's VPC flow logs to determine whether the load balancer is receiving response traffic from the target's IP address. If there's no traffic, then check the target's VPC flow logs or packet captures to determine whether the target received and responded to the load balancer's traffic.

If the load balancer receives the target response traffic but rejects the packets that are going to the client, then check the network ACL's outbound rules. Verify that the rules allow traffic from the ephemeral port (1024-65535) to the client IP address. 

If the Network Load Balancer forwards response traffic to the client but the client doesn't receive it, then check the intermediary devices, such as the firewall. Use TCP-based traceroutes to find the network devices in the network path. Run the following example commands from client device:

 mtr -P  -T  -r
 traceroute -T -p  

Troubleshoot target-related errors

Review the Network Load Balancer's VPC flow logs to identify the target that's receiving the load balancer's traffic, and verify that the traffic reached the target. If the target group activated client IP address preservation, then the client's IP address appears in the source IP field. If client IP address preservation is deactivated, then you see the load balancer's private IP address.

If the target's VPC flow logs show that the target rejected the incoming traffic from the load balancer, then check the target's security group and network ACL. Make sure that the target's port in the target group is open to the load balancer's private IP address or the client IP address. If the logs show that the target rejected outgoing traffic, then check the outbound rules of the target's security group and network ACL. Make sure that the ephemeral port (1024-65535) is open to the load balancer's private IP address or the client IP address.

If SYN packets are coming in but no corresponding SYN_ACK is sent out, then check if OS-level firewalls or proxies, such as iptables, are blocking traffic.

Verify that the target has an application running on the port that's registered to the Network Load Balance's target group. Run commands such as, telnet, CURL, or netcat. For example, if a target is registered on port 80, then log in to the target's terminal and run the following command:

telnet localhost 80
curl -v http://localhost:80  
nc -vz localhost 80

Note: For Microsoft Windows, open a browser on the target and enter http://localhost:80.

Related information

Control traffic to your AWS resources using security groups

Control traffic to subnets using network ACLs

Client IP preservation

Logging IP traffic using VPC Flow Logs