New pods are configured with wrong DNS server
Hi there,
when deploying new pods in my EKS cluster, the DNS server specified in /etc/resolv.conf is not the one configured in "service/kube-dns" in the kube-system namespace. See here:
$ kubectl -n kube-system get svc kube-dns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 42d
But this pod receives a different DNS server:
$ kubectl exec -it busybox -- cat /etc/resolv.conf
nameserver 10.100.0.10
search default.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal
options ndots:5
I've tried to debug this issue but so far no luck. Any help or hints would be greatly appreciated!
Regards,
Khaled
I've had three worker nodes in my EKS cluster, one of them was started by the AWS auto scaler. However, the kubelet was started with the parameter
...
"clusterDNS": [
"10.100.0.10"
],
...
when it should have been started with the parameter
...
"clusterDNS": [
"172.20.0.10"
],
...
The worker node with the wrong parameter also was the one node with the fewest pods scheduled to it so that each new Pod was scheduled on it.
In order to solve the issue, I shut down the node with the incorrect parameter. After shutting down the node, the AWS auto scaler started a new worker node. This new worker node had the correct parameter and the problem I observed was solved.
In order to check on the kubelet parameters, I used this method: https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/#generate-the-configuration-file
Relevant questions
Problem adding nodegroup in EKS cluster with GW NAT
asked a month agoNew pods are configured with wrong DNS server
asked a year agoDNS Lookup Fail with Amazon DNS Server
asked 3 years agoNew to lightsail, domainname
Accepted Answerasked 3 years agoEKS NodeGroup - The aws-auth ConfigMap in your cluster is invalid
asked a year agoforce auto scaling group to scale in by terminating k8s pods ungracefully
asked 10 months agoWorkspace DNS resolution is wrong
asked 5 months agoServer screenshot shows lock screen even when server is not in lockscreen
asked a month agoElastic beanstalk- Server getting created
asked a month agoEKS Cluster stuck in updating
asked 2 months ago