How do I resolve connection timeouts when I connect to my Service that's hosted in Amazon EKS?

Short description

Two of the most common reasons you can't connect to your Service in your Amazon EKS cluster are:

• The security group or network access control list (network ACL) restrictions are preventing traffic from reaching the pod endpoints.
• The Service doesn't select the pod endpoints because the labels don't match.

To resolve these issues, check the security groups and network ACLs that are associated with your worker node instances and load balancer. Also, verify that your Service has the correct labels selected for your pods.

Note: Troubleshooting varies for different service types. The following resolutions are applicable for when you're troubleshooting inaccessible services. To learn more about Kubernetes Service types, see How do I expose the Kubernetes services running on my Amazon EKS cluster?

Resolution

Check your security group and network ACLs

Cluster IP

The cluster IP service type is used for communication between microservices that run in the same Amazon EKS cluster. Make sure that the security group that's attached to the instance where the destination pod is located has an inbound rule to allow communication from the client pod instance.

In most cases, there's a self rule that allows all communication over all ports in the worker node security groups. If you use multiple node groups, each with its own security group, make sure that you allow all communication between the security groups. This lets the microservices that run across the multiple nodes to communicate easily.

To learn more, see Amazon EKS security group considerations.

Node port

The worker node security group should allow incoming traffic on the portthat was specified in the NodePort Service definition. If it's not specified in the Service definition, then the value of the port parameter is the same as the targetPort parameter. The port is exposed on all nodes in the Amazon EKS cluster.

Check the network ACLS that are linked to the worker node subnets. Make sure that your client IP address is on the allow list over the port that the Service uses.

If you're accessing the Kubernetes Service over the internet, make sure that your nodes have a Public IP address. To access the Service, you must use the node's Public IP address and port combination.

Load balancer

Make sure that the load balancer security group allows the listener ports. Also, make sure that the worker node security group allows incoming traffic from the load balancer security group over the port where the application container is running.

If the port that's specified in the Service definition is different from the targetPort, then you must allow the incoming traffic over the port in the worker node security group for the load balancer security group. The port and targetPort are usually the same in the Service definition.

The network ACLs must allow your client IP address to reach the load balancer at the listener port. If you're accessing the load balancer over the internet, then make sure that you created a public load balancer.

Check if your Service selected the pod endpoints correctly

If your pods aren't registered as backends for the Service, then you can receive a timeout error. This can happen when you access the Service from a browser or when you run the curl podIP:podPort command.

Check the labels for the pods and verify that the Service has the appropriate label selectors (from the Kubernetes website).

Run the following commands to verify if your Kubernetes Service correctly selected and registered your pods.

Command:

kubectl get pods -o wide

Example output:

NAME                    READY   STATUS    RESTARTS   AGE       IP                           NODE                         NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-2rtn8   1/1     Running     0       3h4m   172.31.33.214   ip-172-31-33-109.us-west-2.compute.internal       none          none

Command:

kubectl describe svc your_service_name -n your_namespace

Note: Replace your_service_name with your service name and your_namespace with your namespace.

Example output:

Events:            none
Session Affinity:  none
Endpoints:         172.31.33.214:80
....

In the preceding example output, 172.31.33.214 is the pod IP address that was obtained from running the kubectl get pods -o wide command. The 172.31.33.214 IP address also serves as the backend to a service that's running in an Amazon EKS cluster.

---