NLB Endpoint Unreachable

Question

I have an Network Load Balancer that has two availability zone subnets. I have an EC2 instance that is on one of those subnets. However, my NLB endpoint is unreachable even from my EC2.

My NLB has a `TCP:8080` listener and there are two targets in its target group. One target is my EC2 instance port 8080 (my application) and the other target is my EC2 instance port 8000 (my application health check). Both targets are `healthy`.

However, it appears that the NLB's endpoint is not receiving traffic because I cannot even run a ping (ran from my EC2 instance) against my NLB's port; I do not receive any packets.
```
# I'm using the actual ip address for the nlb; this command just hangs.
ping 10.nlb.ip.1
```

When I try a verbose `curl -v ...`, I get something like this:
```
* Host stack-nlb-abc123.elb.us-east-2.amazonaws.com:8080 was resolved.
* IPv6: (none)
* IPv4: 10.nlb.ip.1, 10.nlb.ip.2
*   Trying 10.nlb.ip.1:8080...
* ipv4 connect timeout after 4999ms, move on!
*   Trying 10.nlb.ip.2:8080...
* Connection timed out after 10002 milliseconds
* closing connection #0
curl: (28) Connection timed out after 10002 milliseconds
```

My overall architecture does not rely on my EC2 being able to reach the NLB; I am relying on an API Gateway to reach the NLB. But messages sent to the API Gateway API are failing, so I am trying to troubleshoot the NLB endpoint to see if messages are correctly being sent from the NLB to my EC2 instance. I cannot test my NLB > EC2 workflow unless I can send requests to the NLB.

Note that I am new to AWS and networking.

re:Post Agent · Answer

Based on your description, there are several potential issues that could be causing your Network Load Balancer (NLB) to be unreachable:

1. **Security Group Configuration**: The most common cause of connectivity issues is security group settings. Check that:
- The security group for your EC2 instance allows inbound traffic on ports 8080 and 8000 from the NLB
- If you're using security groups with your NLB, ensure they allow the necessary traffic

2. **Network ACLs**: Even if your security groups are configured correctly, network ACLs associated with your subnets might be blocking traffic. Verify that the network ACLs for your subnets allow both inbound and outbound traffic on the required ports.

3. **Client IP Preservation**: If your NLB target group has the `preserve_client_ip` attribute enabled and your targets include AWS PrivateLink ENIs or another NLB's ENI, this can cause communication loss. Try disabling this attribute if applicable.

4. **Subnet Configuration**: For an internet-facing NLB, ensure it's attached to public subnets that have a route to an internet gateway. If your NLB is attached to private subnets but needs to be internet-facing, this would cause connectivity issues.

5. **Routing**: Verify that the route tables associated with your subnets are correctly configured to allow traffic between the NLB and your EC2 instance.

6. **Health Check Configuration**: While you mentioned your targets are showing as healthy, double-check that the health check configuration is appropriate for your application.

7. **Ping Limitations**: Note that NLBs don't respond to ICMP (ping) requests, so your ping test isn't a valid way to check NLB connectivity. The curl test is more appropriate.

To further troubleshoot:
- Try using tcpdump on your EC2 instance to see if traffic from the NLB is reaching your instance: `tcpdump port 8080`
- Check if your application is actually listening on the specified ports using `netstat -tulpn | grep 8080`
- Review the NLB access logs (if enabled) to see if requests are reaching the NLB

Since your targets are showing as healthy but connections are timing out, this suggests that traffic might be getting blocked somewhere between the NLB and your EC2 instance, most likely due to security group or network ACL configurations.  
**Sources**  
[Troubleshoot a Classic Load Balancer: Client connectivity - Elastic Load Balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/ts-elb-connection-failed.html)  
[NLB Timeout Despite Healthy Targets & Verified Network Path (No VPC Flow Logs for NLB Traffic) | AWS re:Post](https://repost.aws/questions/QUQ1SOLOBVStupFCyx7kflww/nlb-timeout-despite-healthy-targets-verified-network-path-no-vpc-flow-logs-for-nlb-traffic)  
[Troubleshoot a Classic Load Balancer: Health checks - ELB](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/ts-elb-healthcheck.html)  
[ALB Connections Accepted but No Requests Processed – HTTPS Hangs (RequestCount = 0) | AWS re:Post](https://repost.aws/questions/QU7Pw02kKCS6u0R6i8nRxooQ/alb-connections-accepted-but-no-requests-processed-https-hangs-request-count-0)

NLB Endpoint Unreachable

Add your answer

Relevant content