EKS cannot issue certificate to kubelet after node pool creation

Hello,

Greetings for the day!!

From your correspondence I can understand that whenever you are creating an EKS cluster using an assumed role then when you are trying to run the "kubectl exec ..." or "kubectl logs ..." command then you are observing issues. You have also mentioned that you are observing pending csr objects and you need insights into this. Please correct me if I misunderstood anything.

As I do not have access to your EKS cluster, I cannot say what exactly is the configuration or the exact IAM roles, policies configired in your case but I believe I may have figured it out. You can start your troubleshooting by taking a look at the control plane logs and checking the difference between approved and non approved CSR.

You will notice that the the ones that are getting approved will have the fields "responseObject.spec.username" and "user.username" as "system:node:ip-xx-xx-xx-xx.<region-code>.compute.internal"

Where as you will observe that the above value is different for the ones that are not getting approved. You can use the following control plane logs insights query to filter the results:

fields @timestamp, requestObject.spec.signerName, responseObject.metadata.name, responseObject.spec.username, user.username, responseObject.spec.extra.canonicalArn.0, user.extra.sessionName.0
| filter @message like "CertificateSigningRequest"  

By design EKS does not issue certificates for CSRs with signerName "kubernetes.io/kubelet-serving" unless the CSR was actually requested by a kubelet. EKS's custom signer validates this by checking that the requested SANs for CSRs with signerName kubernetes.io/kubelet-serving match an actual EC2 instance's IPs/DNS names.

If the common name does not match then it will fail to approve.

Please refer the below 3 scenarios that we have tested:

Scenario 1: Cluster creator, Cluster role and role attached to the Node Group- All same

The username in this case is kubernetes-admin kubectl logs will error out- remote error: tls: internal error, CSR will fail, as mentioned in previous comments, kubernetes.io/kubelet-serving should match an actual EC2 instance's IPs/DNS names.

Scenario 2: Cluster Creator, Node Role- same ; Cluster role is different

The username is an IAM role , eg: arn:aws:sts::<account-id>:assumed-role/EC2-user/i-<instance-id> kubectl logs will error out- remote error: tls: internal error, CSR will fail - similar reason as above.

Scenario 3: Cluster Creator is different, Cluster role and Node role is the same

The username is EC2 instance's IPs/DNS name eg: system:node:ip-xx-xx-xx-xx.ec2.internal This works as it satisfies the kubernetes.io/kubelet-serving condition.

Based on the above scenarios, it looks like you either belong to scenario 1 or scenario 2 and this is the reason you are observing the said issue.

As per our document https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html "You can't use the same role that is used to create any clusters." Hence, from the console when creating node group you will not see the cluster role, but from cli it is possible like I did in scenario use the same role for cluster role and node role.

Have a fantastic day ahead!!