Why do I get the "webidentityerr" error when using AWS Load Balancer Controller in Amazon EKS?

Short description

You get this error when you use the AWS Load Balancer Controller in Amazon EKS due to the following reasons:

• Incorrect service account configurations
• Incorrect trust relationship of the AWS Identity and Access Management (IAM) role that's used in the service account

With AWS Load Balancer Controller, worker nodes perform the tasks. These worker nodes must be granted access to the AWS Application Load Balancer or AWS Network Load Balancer resources using IAM permissions. You can set up the IAM permissions using IAM roles for the service account. Or, you can attach the IAM permissions directly to the worker node's IAM roles. For more information, see Load Balancer Controller installation on the Kubernetes website.

Resolution

Incorrect service account configurations

To check whether your service account configurations are properly configured, perform the following steps:

1.    Verify the service account name that's defined in your deployment:

kubectl describe deploy <aws-load-balancer-controller> -n kube-system | grep -i "Service Account"

2.    Describe the service account:

kubectl describe sa <aws-load-balancer-controller> -n kube-system

3.    Verify the service account annotation for the IAM role:

Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/<AmazonEKSLoadBalancerControllerRole>

4.    If this annotation is missing or incorrect, then update the annotation. Make sure that you properly associated the IAM role to a service account:

kubectl annotate serviceaccount -n <SERVICE_ACCOUNT_NAMESPACE> <SERVICE_ACCOUNT_NAME> \ eks.amazonaws.com/role-arn=arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE_NAME>

5.    Delete and recreate any existing Amazon EKS pods that are associated with the service account. Be sure to apply the credential environment variables:

kubectl delete pods <aws-loadbalancer-controller-pod> -n <kube-system>

Incorrect trust relationship between the IAM role used and service account

The following examples show some common mistakes that can occur when establishing the trust relationship between your IAM role and service account.

Example 1: IAM role or trust relationship isn't properly defined for the "sts:AssumeRoleWithWebIdentity" action

Verify that the trust relationship is properly defined for the sts:AssumeRoleWithWebIdentity action (and not the sts:AssumeRole action).

The following is an example of a trust relationship that isn't properly defined:

{
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "xxxxx.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
}

To correct the trust relationship, make sure to define the trust relationship for the sts:AssumeRoleWithWebIdentity action:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS-ACCOUNT>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-OIDC-Provider-ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-OIDC-Provider-ID>:sub": "system:serviceaccount:kube-system:<aws-load-balancer-controller-serviceaccount>"
        }
      }
    }
  ]
}

Note: Be sure to replace all variables with your own values.

To use the same IAM role for multiple clusters in one account, define the trust relationship similar to the following:

{
	"Version": "2012-10-17",
	"Statement": [{
			"Effect": "Allow",
			"Principal": {
				"Federated": "arn:aws:iam::<AWS-ACCOUNT>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-Cluster-1-OIDC-Provider-ID>"
			},
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Condition": {
				"StringEquals": {
					"oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-Cluster-1-OIDC-Provider-ID>:sub": "system:serviceaccount: kube-system:<aws-load-balancer-controller-serviceaccount>",
				        "oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-1-OIDC-Provider-ID>:aud": "sts.amazonaws.com"
                                }
			}
		},
		{
			"Effect": "Allow",
			"Principal": {
				"Federated": "arn:aws:iam::<AWS-ACCOUNT>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-Cluster-2-OIDC-Provider-ID>"
			},
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Condition": {
				"StringEquals": {
					"oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-Cluster-2-OIDC-Prob>:sub": "system:serviceaccount: kube-system:<aws-load-balancer-controller-serviceaccount>",
                                        "oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-2-OIDC-Provider-ID>:aud": "sts.amazonaws.com"
				}
			}
		}
	]
}

Example 2: Incorrect OIDC provider ID when creating an Amazon EKS cluster

Make sure to properly create and verify an OIDC provider for your Amazon EKS cluster. Verify that the OIDC provider ID and the associated AWS Region are correctly listed. Otherwise, you receive a WebIdentityErr error.

Example 3: Service account name or its namespace not correctly entered

Make sure to enter the correct service account name and its namespace when you update your AWS Load Balancer Controller deployment.

Example 4: Missing "sts.amazonaws.com" steps from the trust relationship

If the service role associated with your EKS pod is unable to perform the STS operation on the AssumeRoleWithWebIdentity action, then update the trust relationship. The trust relationship must include "sts.amazonaws.com" to perform an STS operation.

For example:

"Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-OIDC-Provider-ID>:sub": "system:serviceaccount:kube-system:<aws-load-balancer-controller-serviceaccount>",
          "oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-OIDC-Provider-ID>:aud": "sts.amazonaws.com"
        }
}

For more information about IAM conditions with multiple keys or values, see Creating a condition with multiple keys or values.

---