Why can't I use an IAM role for the service account in my Amazon EKS pod?

5 minute read
0

I try to use an AWS Identity and Access Management (IAM) role for a service account. My Amazon Elastic Kubernetes Service (Amazon EKS) pod fails to assume the assigned IAM role with an authorization error. Or, my pod tries to use the default IAM role assigned to the Amazon EKS node instead of the IAM role assigned to my pod.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Verify that you have an IAM OIDC identity provider for your Amazon EKS cluster

Create an IAM OIDC provider for your cluster, if you don't already have one. You must have an OIDC identity provider for your cluster to use an IAM role for your service account.

Then, verify that the OIDC identity provider is configured correctly:

  1. Open the IAM console. From the navigation pane, choose Identity providers.
  2. In the Provider column, identify and note the OIDC provider URL.
  3. In a separate tab or window, open the Amazon EKS console. Then, choose Clusters from the navigation pane.
  4. Choose your cluster, and then choose the Configuration tab.
  5. In the Details section, note the value of the OpenID Connect provider URL property.
  6. Verify that the OIDC provider URL from the Amazon EKS console (step 5) matches the OIDC provider URL from the IAM console (step 2).
    If the OIDC provider URL for your Amazon EKS cluster doesn't match any of the OIDC provider URLs in the IAM console, then you must create a new IAM OIDC provider.

Validate your IAM role policies and trust policy configuration

Your IAM role might not have the full range of permissions needed. Your IAM role's trust relationship policy could also have syntax errors, if you created your IAM role with the AWS Management Console or AWS CLI.

To validate your IAM role policies and check for syntax errors in your trust policy, do the following:

  1. Open the IAM console.
  2. In the navigation pane, choose Roles, and then choose your role.
  3. Choose the Permissions tab on your role's page, and then verify that all your required permissions are assigned to the role.
  4. Choose the Trust Relationships tab, and then choose Edit trust relationship.
  5. In the policy document for your trust relationship, verify that the format of your policy matches the format of the following JSON policy:
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "arn:aws:iam::your-account-id:oidc-provider/oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER:sub": "system:serviceaccount:your-namespace:your-service-account",
              "oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER:aud": "sts.amazonaws.com"
            }
          }
        }
      ]
    }
    In your JSON policy, review the format of the Federated property line and the StringEquals property line. In the Federated line, confirm that your AWS Region code (your-region-code), account ID (your-account-id), and unique OIDC identifier (EXAMPLE_OIDC_IDENTIFIER) are formatted correctly. In the StringEquals line, confirm that your Region code (your-region-code), OIDC unique identifier (EXAMPLE_OIDC_IDENTIFIER), Kubernetes namespace (your-namespace), and Kubernetes service account name (your-namespace) are formatted correctly.
  6. If you edit your policy document to correct formatting errors, then choose Update Trust Policy.

Confirm that your service account exists and has a properly formatted annotation for the IAM role's ARN

  1. Confirm that your Kubernetes service account exists:

    $ kubectl get serviceaccount YOUR_ACCOUNT_NAME -n YOUR_NAMESPACE -o yaml

    Note: Replace YOUR_ACCOUNT_NAME with your account name. Replace YOUR_NAMESPACE with your namespace.
    If the preceding command doesn't return a service account name, then create a service account. For more information, see Use more than one ServiceAccount on the Kubernetes website.

  2. Confirm that your service account has the name that you expect. Also, confirm that its role-arn annotation is correctly formatted. For example:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::012345678912:role/my-example-iam-role
      name: my-example-serviceaccount
      namespace: my-test-namespace

Use a test pod to verify that the service account works

Run a test pod to verify that the service account works correctly. Then, check if the pod can mount environment variables correctly and can assume the specified IAM role.

  1. Create a local YAML file called awscli-pod.yaml. For example:

    apiVersion: v1
    kind: Pod
    metadata:
      name: awscli
      labels:
        app: awscli
    spec:
      serviceAccountName: YOUR_SERVICE_ACCOUNT
      containers:
      - image: amazon/aws-cli
        command:
          - "sleep"
          - "604800"
        imagePullPolicy: IfNotPresent
        name: awscli
      restartPolicy: Always

    Note: Replace YOUR_SERVICE_ACCOUNT with your Kubernetes service account name.

  2. Create the test pod (from the YAML file) in your namespace:

    $ kubectl apply -f ./awscli-pod.yaml -n YOUR_NAMESPACE

    Note: Replace YOUR_NAMESPACE with your namespace.

  3. Confirm that the awscli pod has the correct environment variables:

    $ kubectl exec -n YOUR_NAMESPACE awscli -- env | grep AWS

    The output looks similar to the following:

    AWS_ROLE_ARN=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME
    AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
  4. Confirm that the test pod is with the correct IAM role:

    $ kubectl exec -it awscli -n YOUR_NAMESPACE -- aws sts get-caller-identity

    The output looks similar to the following:

    {
        "UserId": "REDACTEDY471234567890:botocore-session-1632772568",
        "Account": "012345678912",
        "Arn": "arn:aws:sts::012345678912:assumed-role/your-iam-role/botocore-session-1632772568"
    }

    Note the Arn value, including the IAM role name that you receive in this output.

  5. After you verify the IAM role, delete the awscli pod:

    $ kubectl delete -f ./awscli-pod.yaml -n YOUR_NAMESPACE

    If the awscli pod shows the correct IAM role, then the IAM roles for service accounts feature work correctly.

The preceding steps confirm that the IAM token is correctly mounted to the pod. If your application still can't use the token file correctly, then there might be an issue at the application or SDK level. This issue can be related to how the application ingests AWS credentials or because the SDK version isn't supported. For more information, see Using the Default Credential Provider Chain, Credentials on the Boto3 website, and Using a supported AWS SDK.

AWS OFFICIAL
AWS OFFICIALUpdated 4 months ago
2 Comments

I was tearing my hair out on this one. I had two pods with the same service account setup, but one of them would consistently return the node group IAM role instead of the service role when I ran "aws sts get-caller-identity". After a lot of debugging, I noticed that the pod that worked was running AWS CLI v2, while the pod that didn't work had AWS CLI v1. Apparently something about version 1 doesn't work with the EKS pod environment variables and the web identity token.

replied 4 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 4 months ago