How do I troubleshoot unhealthy targets for Network Load Balancers in Amazon EKS?

Short description

The targets for your Network Load Balancer could be unhealthy for the following reasons:

• The health check has an incorrect configuration.
• There's an unexpected exception from the Pod.
• A custom Amazon Virtual Private Cloud (Amazon VPC) DNS on the DHCP options set conflicts with the externalTrafficPolicy setting of Local on the Network Load Balancer.

Resolution

Check if the target group is an IP address or instance

Your Network Load Balancer uses one of two target types:

• The instance target type routes traffic to the worker node NodePort and then through iptables rules to the Pod. This multi-hop path adds latency.
• The IP target type forwards traffic directly to the Pod to reduce latency and simplify troubleshooting. It's a best practice to use the IP target type for Amazon EKS workloads.

Complete the following steps:

1. To check the target type for your service, run the following command:

   kubectl get service SERVICE_NAME -o yaml

   Note: Replace SERVICE_NAME with your service name.
2. Review the output for the service.beta.kubernetes.io/aws-load-balancer-nlb-target-type annotation. If this annotation isn't present, then the default target type is instance.
3. To configure the IP target type, add the following annotation to your Service manifest:

   service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

For more information, see Load Balancing.

Correctly configure the health check

You must determine the Elastic Load Balancing annotations configured for your service. For more information on annotations, see Service on the Kubernetes website.

Complete the following steps:

1. To get a list of annotations, run the following command:

   kubectl get service SERVICE_NAME -n NAMESPACE -o yaml

   Note: Replace SERVICE_NAME with your service name and NAMESPACE with your namespace.

   Example output:

   service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
   # The number of successive successful health checks required for a backend to be considered healthy for traffic. Defaults to 2, must be between 2 and 10
   
   service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3"
   # The number of unsuccessful health checks required for a backend to be considered unhealthy for traffic. Defaults to 6, must be between 2 and 10
   
   service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "20"
   # The approximate interval, in seconds, between health checks of an individual instance. Defaults to 10, must be between 5 and 300
   
   service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "5"
   # The amount of time, in seconds, during which no response means a failed health check. This value must be less than the service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval value. Defaults to 5, must be between 2 and 60
   
   service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP
   service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
   # can be integer or traffic-port
   
   service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/"
   # health check path for HTTP(S) protocols

2. Verify that the health check annotations match your service configuration to keep targets healthy.

Manually test health checks from a host in Amazon VPC

To simulate a load balancer health check, connect to a host machine that runs in the same Amazon VPC as your cluster. Then, run the appropriate command based on your target type.

Test HTTP/HTTPS health checks for instance target types

For instance target types, run the following curl command:

curl -ivk NODE_IP:NODE_PORT

Note: Replace NODE_IP with the worker node IP address and NODE_PORT with the NodePort assigned to your service. To find the NodePort, run kubectl get service SERVICE_NAME -o wide.

Test HTTP/HTTPS health checks for IP target types

For IP address target types, run the following curl command:

curl -ivk POD_IP:CONTAINER_PORT

Note: Replace POD_IP with the Pod IP address and CONTAINER_PORT with the port where the application listens inside the container. To find the Pod IP, run kubectl get pod POD_NAME -o wide.

Test TCP health checks

If your load balancer uses TCP health checks instead of HTTP/HTTPS, use netcat to check TCP connectivity to the Pod.

Complete the following steps:

1. To install netcat, run the following command:

   yum update -y && yum install -y nc

2. To test TCP connectivity to the Pod, run the following command:

   nc -z -v POD_IP CONTAINER_PORT

   Note: Replace POD_IP with your Pod IP address and CONTAINER_PORT with your container port.

   Example command:

   nc -z -v [IP_ADDRESS] 80

   Example output:

   Connection to [IP_ADDRESS] 80 port [tcp/http] succeeded!

Note: Use Reachability Analyzer or the AWS Command Line Interface (AWS CLI) to troubleshoot connectivity issues.

Check for an unexpected exception from the Pod

Instance target type

Complete the following steps:

1. To check the service's health check configuration annotations, run the following command:

   kubectl get service SERVICE_NAME -o yaml

   Note: Replace SERVICE_NAME with your service name. For a full list of annotations, see Health Check on the Kubernetes SIGs website.

2. To verify that Pods exist behind the service and check for endpoints, run the following command:

   kubectl get endpoints SERVICE_NAME -o yaml

3. If no endpoints exist, then run the following commands to confirm that the Pod labels match the service selector:

   kubectl describe service SERVICE_NAME
   kubectl get pod --show-labels

4. To confirm that Pods are running without restarts, run the following command:

   kubectl get pods -o wide

5. If Pods show restarts, then run the following commands to retrieve Pod logs and determine the restart cause:

   kubectl logs POD_NAME
   kubectl logs POD_NAME --previous

   Note: Replace POD_NAME with your Pod name. The --previous flag retrieves logs from the last terminated container.

6. From a host machine in your Amazon VPC, run the following curl command to verify that the Pods return the expected HTTP status code:

   curl NODE_IP:NODE_PORT

   Note: Replace NODE_IP with the worker node IP address and NODE_PORT with the NodePort assigned to your service.

   If the command doesn't return the expected HTTP status code, then the backend Pods aren't responding correctly.

7. From the same host machine, run the following curl command to connect directly to the Pod IP address:

   curl POD_IP:CONTAINER_PORT

   Note: Replace POD_IP with the Pod IP address and CONTAINER_PORT with the port where the application listens.

   If the command doesn't return the expected HTTP status code, then the Pod isn't correctly configured.

Note: If the service externalTrafficPolicy is set to Local, then only nodes that run the service backend Pods appear as healthy targets. For more information, see External traffic policy on the Kubernetes website.

IP address target type

Complete the following steps:

1. To check the service's health check configuration annotations, run the following command:

   kubectl get service SERVICE_NAME -o yaml

   Note: Replace SERVICE_NAME with your service name. For a full list of annotations, see Health Check on the Kubernetes SIGs website.

2. From a host machine in your Amazon VPC, run the following curl command to connect to the Pod IP address:

   curl POD_IP:CONTAINER_PORT

   Note: Replace POD_IP with the Pod IP address and CONTAINER_PORT with the port where the application listens.

   If the command doesn't return the expected HTTP status code, then the Pod isn't correctly configured.

Restart the kube-proxy

If the kube-proxy that runs on each node doesn't work correctly, then the kube-proxy might fail to update the iptables rules for the service and endpoints.

To force kube-proxy to recheck and update iptables rules, run the following command to restart it:

kubectl rollout restart daemonset.apps/kube-proxy -n kube-system

Example output:

daemonset.apps/kube-proxy restarted