Why can't I use an IAM role for the service account in my Amazon EKS pod?

Short description

To resolve issues using the IAM roles for service accounts feature, try the following:

• Verify that you have an IAM OpenID Connect (OIDC) identity provider for your Amazon EKS cluster
• Validate your IAM role policies and trust policy configuration
• Confirm that your service account exists and has a properly formatted annotation for the IAM role's ARN
• Use a test pod to verify that the service account is working

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

Resolution

Verify that you have an IAM OIDC identity provider for your Amazon EKS cluster

Create an IAM OIDC provider for your cluster, if you don't already have one. You must have an OIDC identity provider for your cluster to use an IAM role for your service account.

Next, verify that the OIDC identity provider is configured correctly:

1.    Open the IAM console, and then choose Identity providers from the navigation pane.

2.    In the Provider column, identify and note the OIDC provider URL.

3.    In a separate tab or window, open the Amazon EKS console, and then choose Clusters from the navigation pane.

4.    Choose your cluster, and then choose the Configuration tab.

5.    In the Details section, note the value of the OpenID Connect provider URL property.

6.    Verify that the OIDC provider URL from the Amazon EKS console (step 5) matches the OIDC provider URL from the IAM console (step 2).

If the OIDC provider URL for your Amazon EKS cluster doesn't match any of the OIDC provider URLs in the IAM console, then you must create a new IAM OIDC provider.

Validate your IAM role policies and trust policy configuration

Your IAM role might not have the full range of permissions that you expect are assigned to that role. Your IAM role's trust relationship policy could also have syntax errors, if you created your IAM role using the AWS Management Console or AWS CLI.

To validate your IAM role policies and check for syntax errors in your trust policy, do the following:

1.    Open the IAM console.

2.    In the navigation pane, choose Roles, and then choose your role.

3.    Choose the Permissions tab on your role's page, and then verify that all your required permissions are assigned to the role.

4.    Choose the Trust Relationships tab, and then choose Edit trust relationship.

5.    In the policy document for your trust relationship, verify that the format of your policy matches the format of the following JSON policy:

"Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::your-account-id:oidc-provider/oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER:sub": "system:serviceaccount:your-namespace:your-service-account"
        }
      }
    }
  ]
}

In the preceding JSON policy, review the format of the Federated property line and the StringEquals property line. In the Federated line, confirm that your AWS Region code (your-region-code), account ID (your-account-id), and unique OIDC identifier (EXAMPLE_OIDC_IDENTIFIER) are formatted correctly. In the StringEquals line, confirm that your Region code (your-region-code), OIDC unique identifier (EXAMPLE_OIDC_IDENTIFIER), Kubernetes namespace (your-namespace), and Kubernetes service account name (your-namespace) are formatted correctly.

6.    If you edit your policy document to correct formatting errors, then choose Update Trust Policy.

Confirm that your service account exists and has a properly formatted annotation for the IAM role's ARN

1.    Confirm that your Kubernetes service account exists:

$ kubectl get serviceaccount YOUR_ACCOUNT_NAME -n YOUR_NAMESPACE -o yaml

Note: Replace YOUR_ACCOUNT_NAME with your account name. Replace YOUR_NAMESPACE with your namespace.

If the preceding command doesn't return a service account name, then create a service account (from the Kubernetes website).

2.    Confirm that your service account has the name that you expect and that its role-arn annotation is correctly formatted. For example:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::012345678912:role/my-example-iam-role
  name: my-example-serviceaccount
  namespace: my-test-namespace

Use a test pod to verify that the service account is working

You can verify if the service account is working correctly by running a test pod. Then, check if the pod can mount environment variables correctly and can assume the specified IAM role.

Note: You can experience an issue related to your application's credentials even if the IAM roles for service accounts feature work correctly. You can also experience this issue if your pod has the correct environment variables. To prevent this issue, you can use an AWS CLI container image for verification.

1.    Create a local YAML file called awscli-pod.yaml. For example:

apiVersion: v1
kind: Pod
metadata:
  name: awscli
  labels:
    app: awscli
spec:
  serviceAccountName: YOUR_SERVICE_ACCOUNT
  containers:
  - image: amazon/aws-cli
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: awscli
  restartPolicy: Always

Note: Replace YOUR_SERVICE_ACCOUNT with your Kubernetes service account name.

2.    Create the test pod (from the YAML file) in your namespace:

$ kubectl apply -f ./awscli-pod.yaml -n YOUR_NAMESPACE

Note: Replace YOUR_NAMESPACE with your namespace.

3.    Confirm that the awscli pod has the correct environment variables:

$ kubectl exec -n YOUR_NAMESPACE awscli env | grep AWS

Output:

AWS_ROLE_ARN=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

4.    Confirm that the test pod is using the correct IAM role:

$ kubectl exec -it awscli -n YOUR_NAMESPACE -- aws sts get-caller-identity

Output:

{
    "UserId": "REDACTEDY471234567890:botocore-session-1632772568",
    "Account": "012345678912",
    "Arn": "arn:aws:sts::012345678912:assumed-role/your-iam-role/botocore-session-1632772568"
}

Note the Arn value, including the IAM role name (your-iam-role), that you receive in the output from the command in step 4.

5.    Delete the awscli pod after you verify the IAM role:

$ kubectl delete -f ./awscli-pod.yaml -n YOUR_NAMESPACE

If the awscli pod shows the correct IAM role, then the IAM roles for service accounts feature are working correctly.

The preceding steps confirm that the IAM token is correctly mounted to the pod. If your application still can't use the token file correctly, then there is likely an issue at the application or SDK level. This issue is likely related to how the application ingests AWS credentials. For more information, see Using the Default Credential Provider Chain and Credentials (from the Boto3 website).

---