EKS remote error: tls: internal error when running kubectl logs command


I am running the following command: kubectl logs <pod name> and getting:

Error from server: Get "https://x.x.x.x:port/containerLogs/ns/pod/container": remote error: tls: internal error

I am also seeing that all the csr certificates are in pending status, and even after manually approving them, it doesn't help: csr-zlf24 4m59s kubernetes.io/kubelet-serving admin_role <none> Pending

Can you please assist with this?

1 Answer

Hello there, thank you for providing the details.

There can be several reasons on why you get a "remote error: tls: internal error" error. One of the reasons is pending CSR. You can check if there are any pending CSR by this command kubectl get csrAnd, the CSR can be in pending status because of duplicate entries for the node role in aws-auth file.

To check if there are any duplicate entries in aws-auth file you can use this command kubectl get cm aws-auth -n kube-system -o yaml If there are duplicate entries, please follow this document on how to edit aws-auth file and remove duplicate entries - https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html and follow "Edit aws-auth manually" section in the document.

Further, coming to the removal of CSR's as mentioned here -https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/ there is no straight to remove the CSR's but the garbage collection removes CertificateSigningRequests that have not changed state for some duration and pending requests automatically deleted after 24 hours however we can manually deny the certificate using below command and they will be automatically deleted after one hour, but as there are lot of csr in pending it can become hectic to do it but there is no other way and they will be removed after 24 hours.

kubectl certificate deny <certificate-signing-request-name>

However, I would request you to please remove the node role duplicate entry, and then you can try approving the latest csr which is in pending using below command or and then try to run the logs command.

kubectl certificate approve <csr_name>

If you still have issues, please reach out to AWS Premium Support. Thank you.

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions