Running an EKS Auto Mode cluster in a VPC with no direct internet connection.

Sorry for the late reply but it turned out that we managed to make the EKS automode cluster work with the vpc interface endpoints (and hosted zones propagation to the services hostnames from the vpc hosting the VPC interface endpoints). We did not need to configure any http proxy.

I can't give you the exact list of vpc interface endpoints really needed because we already had many (sts,logs,ec2,ecr, etc.) Also we had to host all the container images in ECR repositories in the account. We did not configure the private registry as a proxy to public registries.

Not sure if this is related, I was working on a simple example and got into an similar issue that I created my own VPC(not the default VPC) with two public subnets, and then EKS Auto on that VPC isn't able to pull image, even the metric server comes with it. Then I found out the doc specifically mentioned: If you plan to deploy nodes to a public subnet, the subnet must auto-assign IPv4 public addresses or IPv6 addresses. https://docs.aws.amazon.com/eks/latest/userguide/network-reqs.html#network-requirements-subnets

I would be great the subnet can be auto validated by EKS Auto.

Greeting

Hi Luc,

Thanks for reaching out with such a detailed question! It sounds like you’re navigating a challenging setup with EKS Auto Mode in an environment without direct internet access. This is an edge case, and it’s great that you’re tackling it head-on. Let’s unpack this together and work toward a solution. 😊

---

Clarifying the Issue

You’re trying to run an EKS Auto Mode cluster in a VPC without direct internet connectivity—no Internet Gateway (IGW) or NAT Gateway (NAT GW). You’re using a custom HTTP proxy in another peered VPC, but the nodes in the cluster fail to start, likely because the kubelet on the managed EC2 instances cannot access the API server in AWS's managed VPC.

Previously, you successfully handled this setup using Fargate and Bottlerocket by configuring the necessary HTTPS_PROXY and NOPROXY settings. However, in Auto Mode, you cannot modify the user data for managed nodes, which has limited your control. This restriction is due to the managed nature of EKS Auto Mode, which abstracts infrastructure-level configurations to simplify Kubernetes operations.

Additionally, you’re seeking clarity on which AWS services and endpoints are critical for the EC2 instances to function correctly.

---

Why This Matters

In environments with strict security requirements or no direct internet access, achieving a functional and secure EKS cluster is critical for production workloads. Properly configuring a setup like this ensures compliance with organizational policies while enabling seamless communication between EKS components. However, the added complexity requires careful planning and robust monitoring to avoid operational bottlenecks.

---

Key Terms

• EKS Auto Mode: A simplified configuration of Amazon EKS where AWS manages the cluster and its infrastructure, including worker nodes.
• Kubelet: A Kubernetes component that runs on nodes to ensure containers are running and to communicate with the control plane.
• HTTPS_PROXY/NOPROXY: Environment variables used to define proxy settings for outbound traffic.
• VPC Peering: A networking connection between two VPCs that allows them to route traffic between each other privately.
• VPC Endpoints: Private endpoints in a VPC that allow you to connect to AWS services without using an Internet Gateway.
• Route 53 Private Hosted Zones: A DNS configuration enabling private domain resolution within a specific VPC or set of VPCs.

---

The Solution (Our Recipe)

Steps at a Glance:

1. Identify AWS services and endpoints required for the EKS cluster.
2. Configure VPC endpoints for these services in the VPC without direct internet access.
3. Set up routing between the VPCs using VPC peering and Route 53.
4. Configure DNS resolution to ensure nodes can resolve and access required endpoints.
5. Configure proxy settings if required.
6. Test and validate the cluster setup.

---

Step-by-Step Guide:

1. Identify AWS Service Endpoints
   The managed EC2 instances need access to:
   • EKS Control Plane: For kubelet communication with the API server.
   • Container Registry (ECR): To pull container images.
   • CloudWatch Logs: For logging (optional but recommended).
   • S3: For AWS service metadata and configurations.
   Use the AWS VPC Endpoints documentation to list specific endpoints you’ll need to create.

---

2. Create VPC Endpoints in the Isolated VPC
   In your VPC without internet access, create interface endpoints for:
   • com.amazonaws.region.eks
   • com.amazonaws.region.ecr.api
   • com.amazonaws.region.ecr.dkr
   • com.amazonaws.region.logs
   • com.amazonaws.region.s3
   Example using AWS CLI:

   aws ec2 create-vpc-endpoint --vpc-id vpc-0123456789abcdef \
       --service-name com.amazonaws.us-east-1.eks \
       --vpc-endpoint-type Interface

---

3. Set Up Routing with VPC Peering

   • Ensure routes exist between the peered VPCs to allow traffic from the isolated VPC to the HTTP proxy and other required endpoints.
   • Update route tables to include the HTTP proxy and DNS resolver locations.

4. Configure DNS Resolution

   • Use Route 53 Private Hosted Zones to map service endpoints to the VPC interface endpoints.
   • If the HTTP proxy requires custom DNS, configure the resolver to forward requests for specific domains to the proxy.

   Example DNS Resolution Test:

   dig eks.region.amazonaws.com

   Ensure the output resolves to the private IP address of the VPC endpoint. If the resolution fails, verify that the private hosted zone is correctly associated with your VPC and the appropriate Route 53 records are in place.

---

5. Configure Proxy Settings if Required
   If additional proxy settings are needed, configure these at the VPC or endpoint level as applicable. For example:

   export HTTPS_PROXY=http://proxy-server:port
   export NOPROXY=127.0.0.1,localhost,.yourdomain.com

   Ensure these settings are propagated appropriately to the nodes. Validate them by checking that outbound traffic is routed correctly through the proxy.

---

6. Validate Node Connectivity

   • Use tools like curl or wget on the managed EC2 instances to verify that they can reach required endpoints through the proxy.
   • Check for successful node registration in the Kubernetes API server.

   Example Node Connectivity Test:

   curl -x http://proxy-server:port https://eks.region.amazonaws.com/cluster-name

   A successful test will return an HTTP status code or the expected API server response. If this fails, examine the proxy configuration and VPC route tables for errors.

---

Closing Thoughts

By creating the necessary VPC endpoints, configuring routing, and ensuring proper DNS resolution, you can enable an EKS Auto Mode cluster to operate in a VPC without direct internet access. While this setup introduces additional complexity, it aligns with strict security requirements and maintains functionality.

This approach provides a secure, compliant method for deploying EKS Auto Mode clusters in environments with stringent networking restrictions. However, it requires robust planning, ongoing monitoring, and knowledge of AWS networking tools to maintain operational efficiency.

Here are some resources that might be helpful:

• EKS Cluster VPC Requirements
• VPC Endpoints for Amazon EKS
• Using HTTP Proxies with AWS Services
• Route 53 Private Hosted Zones
• Using ECR in Private VPCs

---

Farewell

I hope this helps you get your EKS Auto Mode cluster up and running in your secure environment, Luc! If you encounter any further issues or need clarification, feel free to ask. Wishing you success with your EKS deployment! 🎉😊

---

Cheers,

Aaron 😊

Hey Luc,

Thanks for the update! I'm glad to hear that you got your EKS Auto Mode cluster working using VPC interface endpoints and Route 53 hosted zone propagation—great troubleshooting work!

It makes sense that you needed multiple VPC endpoints, especially since EKS Auto Mode abstracts away a lot of control over node configurations. Having STS, Logs, EC2, and ECR endpoints in place definitely aligns with what I'd expect for a setup like yours, and hosting all container images in ECR was a smart move to keep everything internal.

It's also interesting that you didn’t need to configure an HTTP proxy at all. That confirms that as long as the right endpoints are in place and properly routed, Auto Mode nodes can function without external internet access. That’s a great insight for others tackling similar setups.

Appreciate you circling back with the results—it’ll definitely help others working through similar challenges. If you ever run into more quirks with EKS Auto Mode in a restricted environment, I’d love to hear how it goes!

Cheers,
Aaron 😊

Hey @dydezira — great observation, and you’re right to dig into subnet configuration. The EKS Auto Mode requirement for public subnets to auto-assign IPv4 or IPv6 addresses can definitely trip up even seasoned AWS users.

Auto Mode abstracts away node provisioning, so it expects the subnets you provide to be ready to support workloads without manual configuration. In the case of public subnets, that means auto-assign public IPs must be enabled. If this isn’t set, nodes won’t get external IPs, can’t reach necessary AWS endpoints (unless routed through VPC endpoints), and the cluster silently fails to bring up workloads or internal agents like metrics-server.

To check or fix this, go to the AWS VPC console → Subnets → pick your subnet → Actions > Modify auto-assign IP settings, and enable auto-assign public IPv4. Alternatively, you can create private subnets, skip public IPs, and then pair the VPC with all necessary Interface VPC Endpoints, as LucD did. That’s the better option for production or restricted environments.

As for your suggestion about validating subnet configurations automatically—totally agree. It’d be a great enhancement if EKS Auto could warn or block at creation time when subnet settings don’t meet basic networking requirements. For now, it’s up to us to double-check those details up front.

Let us know if you’re still stuck on that example setup. Happy to walk through a quick checklist or help you validate the VPC and subnet config. 🛠️😊

---

Cheers, Aaron