How do I resolve the error "You must be logged in to the server (Unauthorized)" when I connect to the Amazon EKS API server?

Short description

You get this error when the AWS Identity and Access Management (IAM) entity that's configured in kubectl isn't authenticated by Amazon EKS. You are authenticated and authorized to access your Amazon EKS cluster based on the IAM entity (user or role) that you use. This error is usually caused by one of the following configurations:

• You configured the kubectl tool to use your IAM user or role.
• You mapped your IAM entity to the aws-auth ConfigMap.

To resolve this issue, complete the steps in one of the following sections based on your use case:

• You're the cluster creator
• You're not the cluster creator
• Use access entries

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

You're the cluster creator

If your IAM entity was used to create the Amazon EKS cluster, then you're the cluster creator. To resolve the error as the cluster creator, complete the following steps:

1. In Amazon CloudWatch Log Insights, select the log group for your Amazon EKS cluster. For example, /aws/eks/my-cluster/cluster. Then, run the following query:

   fields @logstream, @timestamp, @message| sort @timestamp desc
   | filter @logStream like /authenticator/
   | filter @message like "username=kubernetes-admin"
   | limit 50

   Note: Turn on Amazon EKS authenticator logs.
   The query returns the IAM entity that's mapped as the cluster creator:

   @messagetime="2022-05-26T18:55:30Z" level=info msg="access granted" arn="arn:aws:iam::123456789000:user/testuser" client="127.0.0.1:57586" groups="[system:masters]" method=POST path=/authenticate uid="aws-iam-authenticator:123456789000:AROAFFXXXXXXXXXX" username=kubernetes-admin

2. Review the cluster creator IAM entity for the AWS CLI. To see if the IAM entity is configured for AWS CLI in your shell environment, run the following command:

   $ aws sts get-caller-identity

   You can also use a specific profile to run this command:

   $ aws sts get-caller-identity --profile MY-PROFILE

   The output returns the Amazon Resource Name (ARN) of the IAM entity that's configured for AWS CLI.
   Example:

   {
       "UserId": "XXXXXXXXXXXXXXXXXXXXX",
       "Account": "XXXXXXXXXXXX",
       "Arn": "arn:aws:iam::XXXXXXXXXXXX:user/testuser"
   }

   Confirm that the IAM entity that's returned matches the cluster creator IAM entity. If the returned IAM entity isn't the cluster creator, then update the AWS CLI configuration to use the cluster creator IAM entity.

3. To update or generate the kubeconfig file, run the following command:

   $ aws eks update-kubeconfig --name eks-cluster-name --region aws-region

   Note: Replace eks-cluster-name with the name of your cluster. Replace aws-region with the name of your AWS Region.
   To specify an AWS CLI profile for the kubeconfig file, run the following command:

   $ aws eks update-kubeconfig --name eks-cluster-name --region aws-region --profile my-profile

   Note: Replace eks-cluster-name with the name of your cluster. Replace aws-region with the name of your Region. Replace my-profile with the name of your profile.

4. To confirm that the kubeconfig file is updated, run the following command:

   $ kubectl config view --minify

5. To confirm that your IAM entity is authenticated and that you can access your EKS cluster, run the following command:

   $ kubectl get svc

   Example output:

   NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
   kubernetes      ClusterIP   10.100.0.1     <none>        443/TCP   77d

You're not the cluster creator

If your IAM entity wasn't used to create the cluster, then you're not the cluster creator. In this case, complete the following steps to map your IAM entity to the aws-auth ConfigMap to allow access to the cluster:

1. Review the cluster creator IAM entity for the AWS CLI. To see the IAM entity that's configured for AWS CLI in your shell environment, run the following command:

   $ aws sts get-caller-identity

   You can also use a specific profile to run this command:

   $ aws sts get-caller-identity --profile my-profile

   The output returns the ARN of the IAM entity that's configured for AWS CLI.
   Example:

   {
       "UserId": "XXXXXXXXXXXXXXXXXXXXX",
       "Account": "XXXXXXXXXXXX",
       "Arn": "arn:aws:iam::XXXXXXXXXXXX:user/testuser"
   }

   Confirm that the IAM entity that's returned matches the cluster creator IAM entity. If the returned IAM entity isn't the cluster creator, then update the AWS CLI configuration to use the cluster creator IAM entity. Then, use kubectl to retry to access your cluster. If you continue to have the issue, then continue to the next step.

2. If the returned IAM entity isn't the cluster creator, then add your entity to the aws-auth ConfigMap to allow the entity to access the cluster. Because only the cluster admin can modify aws-auth ConfigMap, complete one of the following tasks:
   To use the cluster creator IAM entity to access the cluster, complete the steps in the You're cluster creator section.
   Or, ask the cluster admin to perform this action.

3. To check if your IAM entity is in the aws-auth ConfigMap, run the following command:

   eksctl get iamidentitymapping --cluster cluster-name

   -or-

   kubectl describe configmap aws-auth -n kube-system

   If your IAM entity is in the aws-auth ConfigMap, then continue to the next step. If your IAM entity isn't in the aws-auth ConfigMap, then run the following command to automatically map your IAM entity:

   eksctl create iamidentitymapping \
       --cluster $CLUSTER-NAME \
       --region $REGION \
       --arn arn:aws:iam::XXXXXXXXXXXX:user/testuser \
       --group system:masters \
       --no-duplicate-arns \
       --username admin-user1

   Or, edit the aws-auth ConfigMap to manually map your IAM entity:

   $ kubectl edit configmap aws-auth -namespace kube-system

   To add an IAM entity to the aws-auth ConfigMap, add the IAM user or role ARN to mapUsers.
   IAM user example:

   mapUsers: |
     - userarn: arn:aws:iam::XXXXXXXXXXXX:user/testuser
       username: testuser
       groups:
         - system:masters

   IAM role Example:

   mapRoles: |
     - rolearn: arn:aws:iam::XXXXXXXXXXXX:role/testrole
       username: testrole
       groups:
         - system:masters

   Important:
   The IAM role must be mapped without the path. For more information about rolearn path requirements, see aws-auth ConfigMap does not grant access to the cluster section.
   To specify rolearn for an AWS IAM Identity Center IAM role, remove the path "/aws-reserved/sso.amazonaws.com/REGION" from the Role ARN. Otherwise, the entry in the ConfigMap can't authorize you as a valid user.
   The system:masters group allows superuser access to perform any action on any resource. For more information, see Default roles and role bindings on the Kubernetes website. To restrict access for this user, create an Amazon EKS role and role binding resource. For more information, see Required permissions.

4. Run the following command to update or generate the kubeconfig file. Make sure that the AWS CLI is configured with your IAM entity.

   $ aws eks update-kubeconfig --name eks-cluster-name --region aws-region

   Note: Replace eks-cluster-name with the name of your cluster. Replace aws-region with the name of your Region.
   You can also use a specific profile to run this command:

   $ aws eks update-kubeconfig --name eks-cluster-name --region aws-region --profile my-profile

   Note: Replace eks-cluster-name with the name of your cluster. Replace aws-region with the name of your Region. Replace my-profile with the name of your profile.

5. To confirm that the kubeconfig file is updated, run the following command:

   $ kubectl config view --minify

6. To confirm that your IAM user or role is authenticated, access the cluster again. For example, you can run the following command to confirm that the error is resolved:

   $ kubectl get svc

   Example output:

   NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
   kubernetes      ClusterIP   10.100.0.1     <none>        443/TCP   77d

Use access entries to regain access to the cluster

Use the CreateAccessEntry API to provide or restore access to your Amazon EKS cluster. For more information, see How can I use the Amazon EKS access entry API to recover access to an EKS cluster?

Troubleshooting

When you run a kubectl command, a request is sent to the Amazon EKS cluster API server. Then, the Amazon EKS authenticator tries to authenticate this request. If the Amazon EKS authenticator can't authenticate the request, then check the EKS authenticator logs in CloudWatch. Use the following troubleshooting tips to identify the issue.

Access your EKS authenticator logs

1. Turn on logging for your Amazon EKS cluster.
2. Open CloudWatch Log Insights.
3. Select the log group for your cluster. Example: "/aws/eks/example-cluster/cluster".
4. Run the following query:

   fields @timestamp, @message| filter @logStream like /authenticator/
   | sort @timestamp desc
   | limit 1000

   To identify log lines for the same time interval when you got the error, run kubectl commands.

Review your EKS authenticator logs

Based on the cause of the error, complete the following tasks:

• If the issue is from the use of an incorrect IAM entity for kubectl, then review the kubectl kubeconfig and AWS CLI configuration. Make sure that you use the correct IAM entity. In the following output example, the IAM entity that kubectl used can't be validated. Review the IAM entity that kubectl uses and make sure that the entity exists in IAM and the entity's programmatic access is turned on.
  Example output:

  time="2022-12-26T20:46:48Z" level=warning msg="access denied" client="127.0.0.1:43440" error="sts getCallerIdentity failed: error from AWS (expected 200, got 403). Body: {\"Error\":{\"Code\":\"InvalidClientTokenId\",\"Message\":\"The security token included in the request is invalid.\",\"Type\":\"Sender\"},\"RequestId\":\"a9068247-f1ab-47ef-b1b1-cda46a27be0e\"}" method=POST path=/authenticate

• If the issue because your IAM entity isn't mapped or is incorrectly mapped in aws-auth ConfigMap, then review the aws-auth ConfigMap. Make sure that the IAM entity is correctly mapped and meets the requirements that are listed in the You're not cluster creator section. For incorrectly mapped or missing IAM entities, the EKS authenticator logs look similar to the following output example:

  time="2022-12-28T15:37:19Z" level=warning msg="access denied" arn="arn:aws:iam::XXXXXXXXXX:role/admin-test-role" client="127.0.0.1:33384" error="ARN is not mapped" method=POST path=/authenticate

• If the aws-auth ConfigMap was updated and you lost access to the cluster, then use the cluster creator IAM entity to access the cluster. Because the cluster creator doesn't need to be mapped in the aws-auth ConfigMap, use this IAM entity to access the cluster.

• If the cluster creator IAM entity was deleted, then recreate the same IAM entity with the same naming convention. Then, this recreated cluster creator IAM entity can have the same ARN as the IAM entity. Then, complete the steps in the You're the cluster creator section to use the IAM entity to access the cluster.

• If the cluster creator is an IAM role that was created for an SSO user that was removed, then you can't recreate this IAM role. In this case, reach out to AWS Support for assistance.

Related information

How do I provide access to other IAM users and roles after cluster creation in Amazon EKS?

Using RBAC authorization on the Kubernetes website

Grant IAM users access to Kubernetes with EKS access entries

EKS Access Entries on the eksctl website