By using AWS re:Post, you agree to the Terms of Use

AWS EKS Internal Network load balancer (NLB) timed out while trying to connect from underlying eks worker node

0

I tried following the steps from this blog post https://www.middlewareinventory.com/blog/internal-external-load-balancer-aws-eks/#Creating_AWS_Internal_Load_Balancer_with_K8s_Service_EKS to setup and test my internal NLB deployed on my eks cluster and I am able to successfully test it out. The NLB service is accessible from both windows and linux machine residing in the same VPC as my eks cluster. But when I tried to curl this NLB via underlying worker node, I am getting the time out as shown below. Is this a bug or that's how NLB in EKS clusters are designed? can someone help me to improve as this is 1st time I am dealing with internal NLB

curl -v http://xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com/SampleWebApp
*   Trying xxx.xx.xx.xx:80...
* connect to xxx.xx.xx.xx port 80 failed: Connection timed out
* Failed to connect to xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com port 80 after 130915 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com port 80 after 130915 ms: Connection timed out
  • Hi @Venkat,

    I am also experiencing the same issue. Did anybody got a resolution yet? Please share your experience.

    Thanks, Mithun

1 Answers
0

Hello,

I believe the issue you are facing could be related to the client IP preservation setting on your Network Load Balancer.

When you are trying to reach your kubernetes service from the target instance of your NLB and if the request is being routed by the NLB to the same target instance, the client IP and the destination IP are the same for that request. Therefore, the request will not go through due to the hairpinning effect. Please refer this for more info.

If you disable client IP preservation, the private IP address of the NLB becomes the client IP address for all incoming traffic, and thereby remediating your issue.

You can use "IP" targets instead of "instance" targets to remediate this issue. You can do this by modifying the service.beta.kubernetes.io/aws-load-balancer-type annotation in your service manifest to nlb-ip instead of nlb.

However, if you want the keep using "instance" targets, you can disable client IP preservation setting by adding the below annotation to your service manifest.

service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false

I hope this helps!

profile picture
SUPPORT ENGINEER
answered 23 days ago
  • thanks, @Venkat_P, let me try these approach and come back in case of issues

  • Hello @Venkat_P, tried applying this service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false annotation but it does not solve the problem and still getting time out while trying to curl NLB via underlying node

  • also tried the other approach : "You can use "IP" targets instead of "instance" targets to remediate this issue. You can do this by modifying the service.beta.kubernetes.io/aws-load-balancer-type annotation in your service manifest to nlb-ip instead of nlb" but after this no EXTERNAL-IP shows as <pending> for my sample service. how to test then?

  • I am sorry that it didn't work for you. Can you roll it back to 'nlb' and try to manually set the preserve_client_ip.enabled=false on the target-group attributes and see if that is resolving your problem. If that does, then we can focus on why it is not working with the annotations. Thanks!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions