Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

EKS Access Entry for Managed Nodes existing but MNG nodes cannot join the EKS cluster

0

Hello, When we upgraded our EKS cluster to K8S 1.31 version we switched from EKS authentication mode CONFIG_MAP to API_AND_CONFIG_MAP (in Mar 2025). We noticed an EKS Access Entry was created automatically for the managed node group for the IAM principal ARN arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes, type: EC2 Linux, system:node:{{EC2PrivateDNSName}}, K8S Group system:nodes .

Now we are upgrading to EKS 1.32. The control plane was upgrade last week and today we upgraded the worker nodes. We are using IaC terraform modules + terragrunt to perform this upgrade. Since we noticed the existence of an EKS access entry for the IAM role assigned to the EKS Managed Node Groups, we said to set the terraform variable var.add_iam_role_mapping = false. If this variable is set to false, it will not be created a K8S object type configmap in the namespace = aws-auth-merger for the IAM role of the EKS managed node group's IAM role "IAM-role-EKS-Managed-Node-Group-Nodes".

This is the K8S config map

module.eks_iam_role_mapping[0].kubernetes_config_map.eks_to_k8s_role_mapping will be created

  • resource "kubernetes_config_map" "eks_to_k8s_role_mapping" {

    • data = {

      • "mapRoles" = <<-EOT - "groups": - "system:bootstrappers" - "system:nodes" "rolearn": "arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes" "username": "system:node:{{EC2PrivateDNSName}}" EOT
      • "mapUsers" = jsonencode([]) }

    • id = (known after apply)

    • metadata {

      • generation = (known after apply)
      • labels = {
        • "eks-cluster" = "my-cluster-eks01" }
      • name = "apps"
      • namespace = "aws-auth-merger"
      • resource_version = (known after apply)
      • uid = (known after apply) } }

But when we started the managed nodes upgrade, we noticed that the worker nodes could not attach to the EKS cluster. The EKS access entry exists but the config map above was deleted by us (since we said there is already an EKS access entry for the IAM role of the EKS managed node group "IAM-role-EKS-Managed-Node-Group-Nodes"). Checking the EKS audit logs between 16:05 - 16:50 UTC+3, we found using this filter in Log Insights:

fields @timestamp, @message | filter @logStream like /authenticator/ | filter @message like /denied/ or @message like /failed/ or @message like /error/ | sort @timestamp desc | limit 50

the logs:

43

2025-08-18T16:51:11.527+03:00 time="2025-08-18T13:51:11Z" level=warning msg="access denied" arn="arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes" client="127.0.0.1:51720" error="Identity is not mapped" method=POST path=/authenticate

Log pattern: Token-1 level=warning msg="access denied" Token-2 client=client-3:Token-4 error="Identity is not mapped" method=POST path=/authenticate 100%

Other logs examples:

@message

@timestamp 2025-08-18T16:20:36.472+03:00 time="2025-08-18T13:20:36Z" level=info msg="STS response" accesskeyid=accountid=<aws-account-id> arn="arn:aws:sts::<aws-account-id>:assumed-role/IAM-role-EKS-Managed-Node-Group-Nodes/i-042fdf868868cebd2" client="127.0.0.1:51720" method=POST path=/authenticate session=i-042fdf868868cebd2 stsendpoint=sts.us-east-1.amazonaws.com userid=Field 1 Value @aws.account <aws-account-id> @aws.region us-east-1 @ingestionTime 1755523241230 @log <aws-account-id>:/aws/eks/my-cluster-eks01/cluster @logStream authenticator-02c4f6f03edc0a7bc925a6d643aa1b80 @message time="2025-08-18T13:20:36Z" level=info msg="STS response" accesskeyid=accountid=<aws-account-id> arn="arn:aws:sts::<aws-account-id>:assumed-role/IAM-role-EKS-Managed-Node-Group-Nodes/i-042fdf868868cebd2" client="127.0.0.1:51720" method=POST path=/authenticate session=i-042fdf868868cebd2 stsendpoint=sts.us-east-1.amazonaws.com userid= @timestamp 1755523236472 2025-08-18T16:20:36.472+03:00 time="2025-08-18T13:20:36Z" level=warning msg="access denied" arn="arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes" client="127.0.0.1:51720" error="Identity is not mapped" method=POST path=/authenticate Field 2 Value @aws.account <aws-account-id> @aws.region us-east-1 @ingestionTime 1755523241230 @log <aws-account-id>:/aws/eks/my-cluster-eks01/cluster @logStream authenticator-02c4f6f03edc0a7bc925a6d643aa1b80 @message time="2025-08-18T13:20:36Z" level=warning msg="access denied" arn="arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes" client="127.0.0.1:51720" error="Identity is not mapped" method=POST path=/authenticate @timestamp 1755523236472

As we said there is an EKS access entry for the IAM role of the EKS Manage node Group:

{ "accessEntries": [ .... "arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes", ...." ] } (END)

aws eks describe-access-entry --cluster-name my-cluster-eks01 --region us-east-1 --principal-arn arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes

{ "accessEntry": { "clusterName": "my-cluster-eks01", "principalArn": "arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes", "kubernetesGroups": [ "system:nodes" ], "accessEntryArn": "arn:aws:eks:us-east-1:<aws-account-id>:access-entry/my-cluster-eks01/role/<aws-account-id>/IAM-role-EKS-Managed-Node-Group-Nodes/c8c85ded-3b08-6cb1-bdcb-e49f69dafff0", "createdAt": "2024-07-16T17:02:54.921000+03:00", "modifiedAt": "2024-07-16T17:02:54.921000+03:00", "tags": {}, "username": "system:node:{{EC2PrivateDNSName}}", "type": "EC2_LINUX" } }

Then we decided to revert back to the var.add_iam_role_mapping = true so the aws-auth config map for the IAM role of the EKS managed node group to be craeted. After var.add_iam_role_mapping = true and the config map created and successfully merged, then the managed nodes group were upgraded to eks 1.32 and the nodes attached successfully to the cluster. Please see the logs below, " msg="access granted"". Field Value @aws.account <aws-account-id> @aws.region us-east-1 @ingestionTime 1755530494494 @log <aws-account-id>:/aws/eks/my-cluster-eks01/cluster @logStream authenticator-f60308a91a886bf1921e814db231d34d @message time="2025-08-18T15:21:25Z" level=info msg="access granted" arn="arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes" client="127.0.0.1:38908" groups="[system:bootstrappers system:nodes]" method=POST path=/authenticate stsendpoint=sts.us-east-1.amazonaws.com uid="aws-iam-authenticator:<aws-account-id>:" username="system:node:ip-10-21-180-192.ec2.internal" @timestamp 1755530489464

During this upgrade we also switched from EKS worker nodes with AMI Amazon Linux 2 to Amazon Linux 2023.

Our MN Groups are: my-cluster-eks01-SPOT-apps-apps-0 my-cluster-eks01-SPOT-apps-apps-1 my-cluster-eks01-SPOT-apps-apps-2

Our question: Why new nodes failed to attach to the cluster when there is existing EKS access entry for the IAM role "arn:aws:iam::<aws-account-id>:role/IAM-role-EKS-Managed-Node-Group-Nodes"?

Thank you.

Topics
Containers
Tags
Amazon Elastic Kubernetes Service
Language
English
asked 9 months ago386 views
1 Answer
0

When upgrading your EKS cluster from 1.31 to 1.32 with the authentication mode set to API_AND_CONFIG_MAP, you encountered an issue where your managed nodes couldn't join the cluster despite having an EKS Access Entry for the node IAM role. This happened because you set var.add_iam_role_mapping = false, which prevented the creation of the necessary ConfigMap in the aws-auth-merger namespace.

The root cause of this issue is that during the transition period between authentication modes, both mechanisms need to be in place. Even though you have an EKS Access Entry created for your managed node group's IAM role, the ConfigMap entry is still required when using the API_AND_CONFIG_MAP authentication mode.

The error message in your logs "access denied" with "Identity is not mapped" clearly indicates that the authentication system couldn't map the IAM role to Kubernetes permissions, despite the Access Entry existing. This is because in the API_AND_CONFIG_MAP mode, the system checks both authentication methods, and if either is missing, authentication can fail.

When you reverted to var.add_iam_role_mapping = true and recreated the ConfigMap, the nodes were able to successfully join the cluster, as evidenced by the "access granted" log messages.

This behavior is expected during the transition period to the new EKS authentication system. AWS is gradually moving from the ConfigMap-based authentication to the Access Entry API-based system, but during this transition, both mechanisms need to be maintained for backward compatibility.

For future upgrades, you should maintain both authentication methods until AWS fully transitions to the API-only authentication mode. Once AWS completes the transition to API-only mode, the ConfigMap entries will no longer be necessary.
Sources
aws eks upgreade to 1.32 issue and node can't join in | AWS re:Post
EKS node creation failed | AWS re:Post
Node Group Upgrade Issue After AWS EKS Cluster Upgrade to v1.30 | AWS re:Post
Node Group Upgrade Issue After EKS Cluster Upgrade to v1.30 | AWS re:Post

answered 9 months ago
  • YeniA
    9 months ago

    Related to "The root cause of this issue is that during the transition period between authentication modes, both mechanisms need to be in place. Even though you have an EKS Access Entry created for your managed node group's IAM role, the ConfigMap entry is still required when using the API_AND_CONFIG_MAP authentication mode." is this statement valid also for other IAM roles not belonging to EKS managed node groups ? I mean for my SAML IAM Role, let's say is named IAM-Role-DevOps, there is NO EKS access entry for this role and there is only configmap aws-auth entry, and I can still authenticate to the EKS cluster (list resources, create resources).

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content