Unable to run kubectl & eks commands in a fully private cluster


I have created a VPC fully private (no direct internet access), let's call it VPC-A. This vpc is peer connected to another VPC, let's call it VPC-B. This VPC-B has internet connection and is being used as a gateway for VPC-A.

I have deployed a fully private cluster noly (not any node) in the private subnet of the VPC-A using the guide. The problem is I am not able to run any kubectl and eks command just like mentioned in the guide. After digging a lot on the internet and I found few things to access the cluster. One thing is that I must create a machine in that private VPC and try to access the cluster from there. I also created many issues on github but did not get proper answer. Below are some experts' answers

You can communicate with the K8s API by deploying EC2 instance inside that VPC and defining the EKS K8s API to your kubectl.

Well, I have deployed an instance within the vpc of my cluster but whenever I run the kubectl command from the instance inside the private vpc, I get the following error message Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Also in the EKS fully private cluster guide it is mentioned that

If your setup can reach the EKS API server endpoint via its private address, and has outbound internet access (for EKS:DescribeCluster), all eksctl commands should work.

Can please someone guide me properly that how can I create such setup?

I ran a number of commands to check if anything is wrong with accessing the server address.

nmap -p 443 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com

Starting Nmap 7.80 ( https://nmap.org ) at 2022-09-09 11:11 UTC
Nmap scan report for 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com (192.168.*.*)
Host is up (0.00031s latency).
Other addresses for 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com (not scanned): 192.168.*.*
rDNS record for 192.168.*.*: ip-192-168-*-*.eu-west-*.compute.internal

443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

Another command is

nslookup 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com

Non-authoritative answer:

Name:   1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com
Address: 192.168.*.*
Name:   1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com
Address: 192.168.*.*

And another is

telnet  1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com 443
Trying 192.168.*.*...
Connected to 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com
Escape character is '^]'.
^CConnection closed by foreign hos

It is clear that I can access the api server endpoints from my machine which is in the same vpc as the api server. But still when I run the kubectl command I am getting this output

Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

When I ran the below command

kubectl cluster-info dump

I got the following error message

Unable to connect to the server: proxyconnect tcp: dial tcp: lookup socks5h on server misbehaving


1 Answer
Accepted Answer

The last error message seems to be referring some kind of proxy setup. In general when your EKS API server endpoint is private access only, you need to ensure that the security group attached to EKS cluster allows your connection from the source machine. You also need to configure the kubectl authentication correctly. If you are using IAM access, then you can use AWS CLI to generate the kubeconfig with following command: aws eks update-kubeconfig --name <cluster name> Make sure you have delegated the RBAC access to the IAM user inside your Bastion host before you removed the public access.

More information on how to connect to private only API servers: https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html#private-access

profile pictureAWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions