EKS on VPC using NAT instead of local network

Question

I have an EKS Cluster using VPC and a NAT attached to it.

MY route table is like this for this VPC is like this. The pl-63a5400a is a s3 VPC Endpoint Gateway.

Supposedly all internal traffic when communicating internally should not traffic on NAT gateway. But whenever I try to run a spark application on kubernets I can see a spike on NAT Gateway usage, and so I'm being charged by NAT traffic.

When analyzing VPC Flow Log I can see the ENI of NAT being used for internal communication.

Any advice on that?

Gary Mclean · Answer

When you say internal traffic, do you have an IP for example thats going via the NAT gateway which should be LOCAL?

re:Post Agent · Answer

Based on the information provided, it appears that your EKS cluster is experiencing unexpected NAT Gateway usage for internal communication, which should typically remain within the VPC. Here are some potential reasons and advice to address this issue:

1. Cluster Endpoint Configuration: Check your EKS cluster's API server endpoint configuration. If only the public endpoint is enabled, Kubernetes API requests originating from within your cluster's VPC will leave the VPC and go through the NAT Gateway. To resolve this, enable both public and private endpoints for your cluster. This allows nodes to communicate with the control plane via private VPC endpoints, keeping traffic within the VPC.

2. VPC DNS Settings: Ensure that your VPC has `enableDnsHostnames` and `enableDnsSupport` set to `true`, and the DHCP options for your VPC include `AmazonProvidedDNS` in its domain name servers list. This is necessary for proper routing of traffic to your API server through the private hosted zone created by EKS.

3. Security Group Configuration: Verify that the control plane security group allows ingress traffic on port 443 from your node security groups. This ensures that nodes can communicate directly with the API server without going through the NAT Gateway.

4. Networking Add-ons: If you're using alternative networking add-ons or self-managing the default ones (Amazon VPC CNI, CoreDNS, kube-proxy), ensure they are configured correctly to use the cluster's private endpoint.

5. Application-specific Configuration: Review your Spark application configuration to ensure it's not explicitly using public endpoints for internal communication. Make sure it's configured to use private DNS names or IP addresses for internal services.

6. VPC Endpoints: Consider setting up VPC endpoints for other AWS services your application might be using (besides the S3 endpoint you already have). This can help reduce traffic going through the NAT Gateway.

7. Flow Log Analysis: Continue analyzing your VPC Flow Logs to identify the specific sources and destinations of the traffic going through the NAT Gateway. This can help pinpoint which components of your application or cluster are causing the unexpected external communication.

By addressing these points, you should be able to minimize or eliminate unnecessary NAT Gateway usage for internal communication within your EKS cluster.  
**Sources**  
[VPC and Subnet Considerations - Amazon EKS](https://docs.aws.amazon.com/eks/latest/best-practices/subnets.html)  
[Control network access to cluster API server endpoint - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html)  
[Amazon EKS introduces cluster creation flexibility for networking add-ons - AWS](https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-eks-cluster-creation-flexibility-networking-add-ons/)

EKS on VPC using NAT instead of local network

Add your answer

Relevant content