NLB Connectivity Error
I have a Route 53 A record that points to a Network Load Balancer (NLB-1), which listens on TCP port 4000. NLB-1 forwards traffic to a target group consisting of static IPs belonging to another Network Load Balancer (NLB-2) in a different VPC, which listens on TLS port 4000. NLB-2 then forwards traffic to ECS Fargate tasks, which handle the application workload.
Some clients are experiencing intermittent connectivity errors.
I checked the ECS Fargate task CPUUtilization and MemoryUtilization, which are usually below 40. The NLB-1 has high TCP_Target_Reset_Count, around 95 all the time.
Could you give some tips on how to investigate it? And what metrics should I focus on and how to fix it?
The following are NLB-1 metrics.
- Newest
- Most votes
- Most comments
To investigate the intermittent connectivity errors you're experiencing with your Network Load Balancer (NLB) setup, there are several steps you can take and metrics to focus on:
-
VPC Flow Logs: Enable and analyze VPC Flow Logs for both NLB-1 and NLB-2. This will help you trace the traffic flow and identify where packets might be getting dropped or rejected. Pay special attention to the source and destination IP addresses, ports, and any REJECT entries.
-
Security Groups and Network ACLs: Review the security groups and network ACLs for both NLBs and the Fargate tasks. Ensure that they allow inbound and outbound traffic on the required ports (4000 in this case) between all components of your setup.
-
TCP_Target_Reset_Count: The high TCP_Target_Reset_Count on NLB-1 is concerning. This metric indicates that the targets (in this case, NLB-2) are sending TCP RST packets, which could mean they're rejecting connections. This could be due to several reasons:
- NLB-2 or its targets (Fargate tasks) might be overloaded
- There might be network connectivity issues between NLB-1 and NLB-2
- The TLS configuration on NLB-2 might be causing issues
-
HealthyHostCount and UnhealthyHostCount: Monitor these metrics for both NLBs. If you see fluctuations in these counts, it could indicate issues with the targets.
-
ActiveFlowCount and NewFlowCount: These metrics can help you understand the connection patterns and identify any abnormal spikes or drops.
-
ProcessedBytes: This metric can help you understand the traffic volume being handled by each NLB.
-
TCP_Client_Reset_Count: This metric on NLB-2 could indicate if clients (in this case, NLB-1) are resetting connections.
-
TLS Handshake Errors: Since NLB-2 is using TLS, monitor for any TLS negotiation errors.
-
Packet Captures: Consider performing packet captures on the NLB endpoints to analyze the traffic at a more granular level.
-
Cross-VPC Connectivity: Ensure that the VPC peering or Transit Gateway connecting the two VPCs is properly configured and functioning.
-
Target Group Settings: Review the target group settings for both NLBs, including health check configurations, deregistration delay, and proxy protocol settings.
-
Fargate Task Logs: Although CPU and Memory utilization are low, check the Fargate task logs for any application-level errors that might be causing connection resets.
To fix the issue:
-
If the problem is related to network configuration, adjust security groups, NACLs, or routing tables as necessary.
-
If TLS negotiation is failing, review and update the TLS configuration on NLB-2.
-
If the Fargate tasks are having issues, consider scaling the service or investigating application-level problems.
-
If cross-VPC connectivity is the problem, review and optimize your VPC peering or Transit Gateway setup.
-
Consider implementing connection draining and adjusting deregistration delay settings to handle connections more gracefully during target changes or health check failures.
By systematically investigating these areas and metrics, you should be able to identify the root cause of the intermittent connectivity errors and take appropriate action to resolve them.
Sources
Resolve Network Load Balancer's TCP connectivity errors | AWS re:Post
Troubleshoot Gateway Load Balancer's connectivity issues | AWS re:Post
Troubleshoot your Network Load Balancer - Elastic Load Balancing
