Kubelet certificate issue

This issue with pending kubelet Certificate Signing Requests (CSRs) and TLS errors when running kubectl logs or exec commands is a known problem in EKS clusters. The root cause is likely related to how EKS validates and approves CSRs with the signerName "kubernetes.io/kubelet-serving".

By design, EKS only issues certificates for kubelet-serving CSRs when they are actually requested by a kubelet. EKS's custom signer validates this by checking that the requested Subject Alternative Names (SANs) match an actual EC2 instance's IPs/DNS names.

Based on your symptoms, this could be happening due to one of these scenarios:

1. You might be using the same IAM role for both creating the cluster and for the node group. EKS documentation recommends against using the same role that was used to create clusters for your node groups.

2. The username in the CSR might not match the expected format. For properly working nodes, the username should be in the format "system:node:ip-xx-xx-xx-xx.region.compute.internal".

3. There might be an issue with the hostname configuration in your bootstrap script, which could be causing problems with the node initialization process.

To troubleshoot:

1. Check your control plane logs to see the difference between approved and non-approved CSRs.

2. Verify that you're using different IAM roles for cluster creation and node groups.

3. You can manually approve the pending CSRs using:

kubectl certificate approve <csr-name>

4. Check the kubelet logs on the affected nodes for more specific error messages, which might show "no serving certificate available for the kubelet".

If you're seeing the error "http: TLS handshake error from [IP]: no serving certificate available for the kubelet", this confirms that the certificate issue is preventing proper communication between the kubelet and the API server.
Sources
EKS cannot issue certificate to kubelet after node pool creation | AWS re:Post
Troubleshooting Amazon EKS networking issues at scale in an Enterprise scenario | AWS re:Post