How do I resolve connection timeouts when I connect to my Service that's hosted in Amazon EKS?

Resolution

If your Kubernetes Service's security group or network access control list (network ACL) restricts traffic from your pod endpoints, then you experience connection issues.

To resolve these errors, verify that you correctly configured the security groups and network ACLs associated with your worker node instances and load balancer.

If you use a Network Load Balancer, then verify that you selected the correct labels selected for your pods in Kubernetes.

Run the following command to verify your service configuration:

kubectl describe svc example-service-name -n example-namespace

Note: Replace example-service-name with your service name and example-namespace with your namespace.

Example output:

Name: example-service-name  
 Namespace: example-namespace  
 Labels: example-label-key=example-label-value  
 Annotations: <none>  
 Selector: example-selector-key=example-selector-value  
 Type: ClusterIP IP  
 Family Policy: SingleStack  
 IP: example-cluster-ip  
 IPs: example-cluster-ip  
 Port: example-port-name example-port/TCP  
 TargetPort: example-target-port/TCP  
 NodePort: <unset>  
 Endpoints: example-pod-ip-1:example-port,example-pod-ip-2:example-port  
 Session Affinity: None  
 External Traffic Policy: Cluster  
 HealthCheck NodePort: <unset>  
 Events: example-events

Note your TargetPort and your Endpoints values to complete the following troubleshooting actions.

Check your security group and network ACLs

If your Kubernetes Service is inaccessible, then check your Kubernetes Service type.

If you use one of the following service types, then verify your security groups and network ACLs allow necessary traffic:

• ClusterIP
• NodePort
• LoadBalancer

For more information, see Service type on the Kubernetes website.

ClusterIP

Note: Amazon EKS uses the ClusterIP service type to communicate between microservices in the same cluster.

Verify that the destination pod's security group allows inbound traffic from the client pod. If you use multiple node groups, then allow communication between all node security groups. By default, security groups include a self-referential rule that allows all ports. For more information, see View Amazon EKS security group requirements for clusters.

NodePort

Complete the following steps:

1. Add an inbound rule to your worker node security group to allow traffic on the NodePort service port. If you didn't specify a port, then use the targetPort value instead.
2. Review your worker node subnet network ACLs, and then make sure that they allow traffic from your client IP address and the NodePort range (30000-32767).
3. To access Kubernetes NodePort services from the internet, make sure that your nodes have public IP addresses. Use the format http://node-public-ip:nodeport to reach the service.
   For example, the following is a public IP address:
   http://54.123.45.67:31234
   Note: It's a best practice to allow access only from specific IP ranges instead of 0.0.0.0/0. Make sure that your client IP address is on the allow list over the port that the Service uses.

LoadBalancer

Configure the following security settings:

• Allow inbound traffic on listener ports in the load balancer security group. For example, if your application listens on port 80, then make sure that your inbound rules allow access on port 80.
• Allow outbound traffic to the target port in the load balancer security group. To find your target port, note the targetPort value in your service definition.
  Note: The port where your application pods run is your target port.
• Allow incoming traffic from the load balancer security group to the worker node security group on your application container port.
• If you use different service and target ports, then allow traffic on the target port in the worker node security group. For example, if the targetPort value in your service definition is 80, then your service port must be port 80.
• For Network Load Balancers with client IP preservation, allow client traffic directly to worker nodes.
• Configure load balancer subnet network ACLs to allow traffic from client IP addresses on the listener port for both inbound and outbound rules.
• For internet access, create an internet-facing load balancer instead of an internal one.

Important: It's a best practice to follow the principle of least privilege when you configure security groups and network ACLs.

Confirm that your service selected the pod endpoints correctly

If you don't register your pods as service backends, then you might receive a timeout errors. Make sure that your service's label selectors match your pod labels. For more information, see Labels and selectors on the Kubernetes website.

To verify that you correctly registered your pods, run the following command to check your pod details:

kubectl get pods -o wide

Example output:

           NAME       READY STATUS  RESTARTS AGE       IP                       NODE                      NOMINATED NODE READINESS GATES  
  
nginx-6799fc88d8-2rtn8 1/1  Running     0   3h4m 172.31.33.214 ip-172-31-33-109.us-west-2.compute.internal    none            none

Verify that your pod's IP address matches the service endpoint IP address in your service definition.