AWS EKS Internal Network load balancer (NLB) timed out while trying to connect from underlying eks worker node

Question

I tried following the steps from this blog post https://www.middlewareinventory.com/blog/internal-external-load-balancer-aws-eks/#Creating_AWS_Internal_Load_Balancer_with_K8s_Service_EKS  to  setup and test my internal NLB deployed on my eks cluster and I am able to successfully test it out. The NLB service is accessible from both windows and linux machine residing in the same VPC as my eks cluster. But when I tried to curl this NLB via underlying worker node, I am getting the time out as shown below. Is this a bug or that's how NLB in EKS clusters are designed? can someone help me to improve as this is 1st time I am dealing with internal NLB

```
curl -v http://xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com/SampleWebApp
*   Trying xxx.xx.xx.xx:80...
* connect to xxx.xx.xx.xx port 80 failed: Connection timed out
* Failed to connect to xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com port 80 after 130915 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com port 80 after 130915 ms: Connection timed out

Answer

Hello,

I believe the issue you are facing could be related to the client IP preservation setting on your Network Load Balancer.

When you are trying to reach your kubernetes service from the target instance of your NLB and if the request is being routed by the NLB to the same target instance, the client IP and the destination IP are the same for that request. Therefore, the request will not go through due to the hairpinning effect. Please refer [this](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout) for more info.

If you disable client IP preservation, the private IP address of the NLB becomes the client IP address for all incoming traffic, and thereby remediating your issue.

You can use "IP" targets instead of "instance" targets to remediate this issue. You can do this by modifying the `service.beta.kubernetes.io/aws-load-balancer-type` annotation in your service manifest to `nlb-ip` instead of `nlb`.

However, if you want the keep using "instance" targets, you can disable client IP preservation setting by adding the below annotation to your service manifest.

`service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false`

I hope this helps!

AWS EKS Internal Network load balancer (NLB) timed out while trying to connect from underlying eks worker node

Relevant content