How do I identify the source of connectivity issues with my Network Load Balancer?

Short description

Network Load Balancer connectivity issues can include connection timeouts, TCP reset (RST) packets, or intermittent connection failures. To identify where the problem originates, review your Amazon CloudWatch metrics, Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, access logs, and packet captures.

Resolution

Review CloudWatch metrics to identify the source of connection resets

Open the CloudWatch console and review the following metrics for your Network Load Balancer:

• ActiveFlowCount
• NewFlowCount
• PortAllocationErrorCount
• SecurityGroupBlockedFlowCount_Inbound_TCP
  Note: If you created the Network Load Balancer without a security group, this metric isn't available. Use Amazon VPC Flow Logs instead to identify rejected connections.
• TCP_ELB_Reset_Count
• TCP_Client_Reset_Count
• TCP_Target_Reset_Count

If the TCP reset metrics show that the resets originate from the client or target side, then the issue isn't caused by the load balancer. Take the following actions:

• Investigate the application or server on the target. Review application logs, and then check that the application accepts connections on the expected port.
• Check your resource utilization including CPU, memory, and open file descriptors.
• For client-side resets, review the timeout settings of the client application. Also check the connection pooling configuration, and the network path between the client and the Network Load Balancer.

Turn on and analyze Amazon VPC Flow Logs

To verify whether traffic from clients reaches the Network Load Balancer and to identify rejected connections, use Amazon VPC Flow Logs. To create flow logs, complete the following steps:

1. Open the Amazon Elastic Compute Cloud (Amazon EC2) console.
2. Navigate to Network Interfaces, and filter by the Network Load Balancer name.
3. Get the Elastic Network Interfaces (ENIs) of the Network Load Balancer.
4. Turn on Amazon VPC Flow Logs on the Network Load Balancer.
5. Create a flow log for the ENIs that publishes to Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), or Amazon Data Firehose.
   For Log record format, choose Custom format. Then, choose the following flow log fields to capture detailed TCP connection information: account-id, action, bytes, dstaddr, dstport, end, flow-direction, interface-id, log-status, packets, pkt-dstaddr, pkt-srcaddr, protocol, srcaddr, srcport, start, subnet-id, tcp-flags, type, version, and vpc-id.

After the flow log collects data, check for records with the REJECT status to see if security groups of network access control logs (network ACLs) block traffic. Also, check that the tcp-flags field shows TCP connection states with only the following bitmask values:

• SYN, value 2
• SYN-ACK, value 18
• FIN, value 1
• RST, value 4

Look for records with RST flags to identify whether the client or target initiates connection resets. Then, compare the srcaddr and flow-direction fields to determine whether resets originate from the client-side or target-side traffic.

For more information about available fields and TCP flag values, see Flow log records. For example records that show TCP flag sequences, see Flow log record examples.

Use Network Load Balancer access logs and CloudWatch Logs for detailed connection analysis

Network Load Balancer provides legacy and CloudWatch logging options for TLS listeners that complement Amazon VPC Flow Logs.

Legacy access logs store compressed log files in an Amazon S3 bucket. To turn on legacy access logs, see Enable access logs for your Network Load Balancer.

CloudWatch logs integration delivers access logs to CloudWatch Logs, Firehose, or Amazon S3 with Apache Parquet format. To turn on CloudWatch logs, see CloudWatch logs for your Network Load Balancer.

Note: Both log options are available for TLS listeners only. For TCP and UDP listeners, use Amazon VPC Flow Logs and CloudWatch metrics.

To use Network Load Balancer access logs for troubleshooting, take the following actions.

TLS negotiation failures

The tls_cipher and tls_protocol_version fields in the Network Load Balancer access logs show the cipher suite and TLS version. A value of "-" in these fields indicates the TLS handshake failed before negotiation completed. If you see handshake failures, then verify that the security policy on the TLS listener supports the TLS versions and cipher suites that your clients require. To test what TLS versions and ciphers that the Network Load Balancer accepts, run the following openssl command from the client side:

openssl s_client -connect Network Load Balancer_DNS_NAME:443 -tls1_2 

If the connection fails, then try different TLS versions such as -tls1_3, to identify the versions that the client and Network Load Balancer share. Then update the listener's security policy.

Connection-level errors

The tls_handshake_time field in the Network Load Balancer access logs helps identify TLS handshake latency issues. If tls_handshake_time is unusually high but not "-", then the TLS handshake completed but was slow. A slow handshake can identify issues such as certificate chain validation delays or slow client-side TLS negotiation. If tls_handshake_time displays "-", then the TLS handshake didn't occur or failed. Check that the client established a TCP connection and check the listener's certificate and security policy configuration.

Correlation with client IP addresses

The client_port field in the Network Load Balancer access logs helps trace specific client connections. Filter the logs by this field to determine whether the issue affects a specific client or all clients. If the issue affects only specific clients, then investigate the network path and TLS configuration on those clients. If all clients are affected, then the issue is likely on the Network Load Balancer listener configuration or the targets. Cross-reference the client_port values with Amazon VPC Flow Logs and CloudWatch metrics.

With CloudWatch Logs integration, use CloudWatch Logs Insights queries to analyze traffic patterns in real time and create metric filters for monitoring.

For more information, see Access logs for your Network Load Balancer and CloudWatch logs for your Network Load Balancer.

Perform packet capture analysis

Conduct simultaneous packet captures on both the client and target to check connection reset behavior and identify the side that's initiating connection closures.

On the client end, run the following tcpdump command:

tcpdump -ni any host NLB_IP -w client_capture.pcap 

Note: Replace NLB_IP with the IP address of your Network Load Balancer.

On the target end, the source IP address visible to the target depends on whether you turned on client IP preservation. If you turned on client IP preservation, then the target sees the client's original IP.

To filter by the client IP address, run the following tcpdump command:

tcpdump -ni any host CLIENT_SOURCE_IP -w target_capture.pcap 

Note: Replace CLIENT_SOURCE_IP with the IP address of your client.

If you turn off client IP preservation, then the target sees the private IP address of the Network Load Balancer.

To filter by the Network Load private IP address of the Network Load Balancer, run the following tcpdump command:

tcpdump -ni any host NLB_PRIVATE_IP -w target_capture.pcap 

Note: Replace NLB_PRIVATE_IP with the private IP address of your Network Load Balancer.

To review the source IP addresses and flow direction in the captures to determine what side initiates connection closures, take the following actions:

• Review FIN packets to see a normal connection closure. Check what side sends the first FIN to identify what endpoint closes the connection.
• Review RST packets to see an abnormal connection termination. If the target sends RST immediately after it receives a request, then the application rejects the connection or crashes.
• Review retransmissions to identify packets that are dropping in the network path between the client and the target.
• Review client-side captures that show a complete request while the target-side capture shows no response. Check whether the target application is unresponsive or a firewall is blocking return traffic.
• Compare client-side captures from both sides. If both captures show correct exchanges for the same packets, then investigate target-side factors such as application behavior or resource utilization. If the captures differ, then the issue is likely in the network path between the two capture points.

Related information

How do I troubleshoot connections that fail to establish with my Network Load Balancer?

How do I troubleshoot connections that drop or intermittently fail with my Network Load Balancer?

Troubleshoot your Network Load Balancer

Health checks for Network Load Balancer target groups