AWS Client VPN name resolution for eks endpoint on mac

Question

Hi,

I am trying to get my eks cluster endpoint to be accessible through VPN.   EKS Endpoint is configured to be private/public with a set of IPs on the public side.  On the private side I am connecting to AWS with AWS VPN client which is has proper associations, routes and NO split tunnel as I need to connect to the eks through the private network.  I am also using private dns 10.0.0.2 which is set in the vpn configuration...

I can see configs applied, I am able to access internet and private network throught the VPN, and I can resolve eks control plane hostname correctly to the private IP using nslookup.  However when I use kubectl, it resolves hostname to public IP and fails as my client is not in the set of IPs.  Same thing happens when I use curl to connect to the endpoint.  If I use private IPs of the endpoint I am able to curl and I can run kubectl (if I put host/ip into /etc/hosts file)...

This is happening on mac, the best I can deduce is that somehow my os resolves endpoint to a public IP but I am not able to figure out why 10.0.0.2 dns server is not being used.

I am also getting the same public IP resolution using  dscacheutil -q host -a name *******.eks.amazonaws.com

I finding it hard to believe that I am the only one that would face this type of issue...  Do I need to apply some additional configuration with VPN client?  I remember at my previous job we had a chain of dns servers pushed to all laptops with BIG5 or maybe as OS policies, had some funky behaviors, but makes me wonder IF this was done because VPN client was not able to push DNS server all the way to the top.  And i did flush and kill all etc...

Anton · Answer

this appears to be an issue due to network filters with DNS settings to use https proxy dns lookups.   Proxy will return a public IP since it's not on the private network.

re:Post Agent · Answer

This issue appears to be related to how DNS resolution is working on your Mac when connected to the AWS Client VPN. Even though you've configured the VPN to use the private DNS server (10.0.0.2), your Mac may not be using this DNS server for all lookups.

Here are some troubleshooting steps to resolve this issue:

1. Verify DNS server configuration in your Client VPN endpoint:
Check that the DNS server (10.0.0.2) is properly configured in your Client VPN endpoint. You can verify this using the AWS CLI:
```
aws ec2 describe-client-vpn-endpoints --client-vpn-endpoint-ids your-endpoint-id --query 'ClientVpnEndpoints[0].DnsServers'
```

2. Check DNS resolution from your client:
Test DNS resolution directly using tools like nslookup or dig to see if they're using the VPN's DNS server:
```
nslookup your-eks-endpoint.region.eks.amazonaws.com
dig your-eks-endpoint.region.eks.amazonaws.com
```

3. Security group configuration:
Ensure that security groups allow DNS traffic (UDP/TCP port 53) from the Client VPN CIDR to the DNS servers.

4. DNS resolution order on macOS:
macOS has its own DNS resolution order that might be bypassing the VPN's DNS settings. You can try:
- Checking your DNS settings after connecting to VPN using `scutil --dns`
- Modifying your DNS search order in Network preferences
- Creating a specific DNS configuration for the VPN interface

5. Alternative solution:
Since you mentioned that adding entries to /etc/hosts works, you could create a more permanent solution by:
- Adding the EKS endpoint and its private IP to your /etc/hosts file
- Using a local DNS resolver like dnsmasq to override specific domain resolutions
- Creating a script that updates your hosts file when connecting to the VPN

The issue is likely related to how macOS handles DNS resolution when connected to VPN. The VPN client might not be pushing the DNS configuration with high enough priority to override the system's default DNS resolution path for all applications.  
**Sources**  
[Troubleshooting AWS Client VPN: Tunnel connectivity issues to a VPC - AWS Client VPN](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/VPNTunnelConnectivityTroubleshooting.html)  
[DNS issues with Client VPN connection | AWS re:Post](https://repost.aws/questions/QUvQ-DLHfISRalsTOwGmxaNg/dns-issues-with-client-vpn-connection)  
[Kubernetes concepts for hybrid nodes - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-concepts-kubernetes.html)

Taz · Answer

Hey,

Hope you're keeping well.

On macOS, the AWS Client VPN DNS server you specify (like 10.0.0.2) may not always take precedence because the OS uses `scutil` and `resolver` rules per domain. Even though `nslookup` works, tools like `kubectl` and `curl` rely on the system resolver, which can prefer your default network DNS if the VPN client doesn’t push a per-domain route for `*.eks.amazonaws.com`. In the AWS Client VPN endpoint, make sure **Enable DNS resolution** is turned on and that your associated VPC has `enableDnsHostnames` enabled. You can also add a DNS server mapping in the VPN configuration or on macOS create a `/etc/resolver/eks.amazonaws.com` file pointing to 10.0.0.

Thanks and regards,  
Taz

AWS Client VPN name resolution for eks endpoint on mac

Add your answer

Relevant content