How do I troubleshoot an OIDC provider and IRSA in Amazon EKS?
My pods can't use the AWS Identity and Access Management (IAM) role permissions with the Amazon Elastic Kubernetes Service (Amazon EKS) account token.
Short description
To troubleshoot issues with the OpenID Connect (OIDC) provider and IAM roles for service accounts (IRSA) in Amazon EKS, complete the steps in one of the following sections:
- Check if you have an existing IAM OIDC provider for your cluster
- Check if your IAM role has an attached needful IAM policy with required permissions
- Verify that the IAM role trust relations are correctly set
- Check if you created a service account
- Verify that the service account has the correct IAM role annotations
- Verify that you correctly specified the serviceAccountName in your pod
- Check the environment variables and permissions
- Verify that the application uses a supported AWS SDK
- Check the pod user and group
- Recreate pods
- Verify that the audience is correct
- Verify that you configured the correct thumbprint
- For the AWS China Region, check the AWS_DEFAULT_REGION environment variable
Resolution
Check if you have an existing IAM OIDC provider for your cluster
If a provider already exists, then you receive an error that's similar to the following message:
"WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E\n\tstatus code: 400"
1. Check your cluster's OIDC provider URL:
$ aws eks describe-cluster --name cluster_name --query "cluster.identity.oidc.issuer" --output text
See the following example output:
https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E
2. List the IAM OIDC providers in your account. Replace EXAMPLED539D4633E53DE1B716D3041E (include < >) with the value returned from the previous command:
aws iam list-open-id-connect-providers | grep EXAMPLED539D4633E53DE1B716D3041E
See the following example output:
"Arn": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
If the preceding command returns an output, then you already have a provider for your cluster. If the command doesn't return an output, then you must create an IAM OIDC provider.
Check if your IAM role has an attached needful IAM policy with required permissions
1. Open the IAM console.
2. In the navigation pane, choose Roles.
3. Choose the role that you want to verify.
4. Under the Permissions tab, verify if this role has the required policy attached.
Verify that the IAM role trust relations are correctly set
With the AWS Management Console:
1. Open the IAM console.
2. In the navigation pane, choose Roles.
3. Choose the role that you want to check.
4. Choose the Trust Relationships tab to verify that the format of your policy matches the format of the following JSON policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME", "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud": "sts.amazonaws.com" } } } ] }
To verify trust relations, run following command with your role name in the AWS Command Line Interface (AWS CLI):
$ aws iam get-role --role-name EKS-IRSA
Note: Replace EKS-IRSA with your IAM role name.
In the output JSON, look for the AssumeRolePolicyDocument section.
See the following example output:
{ "Role": { "Path": "/", "RoleName": "EKS-IRSA", "RoleId": "AROAQ55NEXAMPLELOEISVX", "Arn": "arn:aws:iam::ACCOUNT_ID:role/EKS-IRSA", "CreateDate": "2021-04-22T06:39:21+00:00", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud": "sts.amazonaws.com", "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME" } } } ] }, "MaxSessionDuration": 3600, "RoleLastUsed": { "LastUsedDate": "2021-04-22T07:01:15+00:00", "Region": "AWS_REGION" } } }
Note: Check that you have specified the correct AWS Region, Kubernetes service account name, and Kubernetes namespace.
Check if you created a service account
Use the following command:
$ kubectl get sa -n YOUR_NAMESPACE
Note: Replace YOUR_NAMESPACE with your Kubernetes namespace.
See the following example output:
NAME SECRETS AGE default 1 28d irsa 1 66m
If you don't have a service account, see Configure service accounts for pods (from the Kubernetes website).
Verify that the service account has the correct IAM role annotations
Use the following command:
$ kubectl describe sa irsa -n YOUR_NAMESPACE
Note: Replace irsa with your Kubernetes service account name. Replace YOUR_NAMESPACE with your Kubernetes namespace.
See the following example output:
Name: irsa Namespace: default Labels: none Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME Image pull secrets: none Mountable secrets: irsa-token-v5rtc Tokens: irsa-token-v5rtc Events: none
Verify that you correctly specified the serviceAccountName in your pod
Use the following command:
$ kubectl get pod POD_NAME -o yaml -n YOUR_NAMESPACE| grep -i serviceAccountName:
Note: Replace POD_NAME and YOUR_NAMESPACE with your Kubernetes pod and namespace.
See the following example output:
serviceAccountName: irsa
Check the environment variables and permissions
Look for AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE in the pod's environment variables:
$ kubectl -n YOUR_NAMESPACE exec -it POD_NAME -- env | grep AWS
See the following example output:
AWS_REGION=ap-southeast-2 AWS_ROLE_ARN=arn:aws:iam::111122223333:role/EKS-IRSA AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token AWS_DEFAULT_REGION=ap-southeast-2
Verify that the application uses a supported AWS SDK
The SDK version must be greater than or equal to the following values:
Java (Version 2) — 2.10.11 Java — 1.11.704 Go — 1.23.13 Python (Boto3) — 1.9.220 Python (botocore) — 1.12.200 AWS CLI — 1.16.232 Node — 3.15.0 Ruby — 2.11.345 C++ — 1.7.174 .NET — 3.3.659.1 PHP — 3.110.7
To check for latest supported SDK version, see Using a supported AWS SDK.
Check the pod user and group
Use the following command:
$ kubectl exec -it POD_NAME -- id uid=0(root) gid=0(root) groups=0(root)
Note: By default, only containers that run as root have the proper file system permissions to read the web identity token file.
If your containers aren't running as root, then you can receive the following errors:
Error: PermissionError: [Errno 13] Permission denied: '/var/run/secrets/eks.amazonaws.com/serviceaccount/token
-or-
WebIdentityErr: failed fetching WebIdentity token: \ncaused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied
To provide the proper file system permissions, make sure that your containers run as root. For clusters 1.18 or lower, provide the following security context for the containers in your manifest:
apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: template: metadata: labels: app: my-app spec: serviceAccountName: my-app containers: - name: my-app image: my-app:latest securityContext: fsGroup: 1337 ...
Note: The fsGroup ID is arbitrary. You can choose any valid group ID. The preceding security context setting is not required for clusters 1.19 or later.
Recreate pods
If you created pods before you applied IRSA, then recreate the pods.
See the following example command:
$ kubectl rollout restart deploy nginx
See the following example output:
deployment.apps/nginx restarted
For daemonsets or statefulsets deployments, you can use the following command:
$ kubectl rollout restart deploy DEPLOYMENT_NAME
If you have created only one pod, then you must delete the pod and recreate it.
See the following example command to delete the pod:
$ kubectl delete pod POD_NAME
See the following example command to recreate the pod:
$ kubectl apply -f SPEC_FILE
Note: Replace SPEC_FILE with your Kubernetes manifest file path and file name.
Verify that the audience is correct
If you created the OIDC provider with the incorrect audience, then you receive the following error:
Error - An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Incorrect token audience
Check the IAM identity provider for your cluster. Your ClientIDList is sts.amazonaws.com:
$ aws iam get-open-id-connect-provider --open-id-connect-provider-arn arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E
See the following example output:
{ "Url": "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E", "ClientIDList": [ "sts.amazonaws.com" ], "ThumbprintList": [ "9e99a48a9960b14926bb7f3b02e22da2b0ab7280" ], "CreateDate": "2021-01-21T04:29:09.788000+00:00", "Tags": [] }
Verify that you configured the correct thumbprint
If the thumbprint that's configured in the IAM OIDC is not correct, you can receive the following error:
failed to retrieve credentials caused by: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint
To automatically configure the correct thumbprint, use eksctl or the AWS Management Console to create the IAM identity provider. For other ways to obtain a thumbprint, see Obtaining the thumbprint for an OpenID Connect identity provider.
For the AWS China Region, check the AWS_DEFAULT_REGION environment variable
If you use IRSA for a pod or daemonset that's deployed to a cluster in the AWS China Region, then set the AWS_DEFAULT_REGION environment variable in the pod specification. If you don't, the pod or daemonset can receive the following error:
An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid
Use the following example to add the AWS_DEFAULT_REGION environment variable to your pod or daemonset specification:
apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: template: metadata: labels: app: my-app spec: serviceAccountName: my-app containers: - name: my-app image: my-app:latest env: - name: AWS_DEFAULT_REGION value: "AWS_REGION" ...
Related videos

Relevant content
- asked 6 months agolg...
- asked 3 months agolg...
- Accepted Answerasked 8 months agolg...
- asked a year agolg...
- asked 6 months agolg...
- AWS OFFICIALUpdated 2 months ago
- AWS OFFICIALUpdated 2 months ago
- AWS OFFICIALUpdated 4 months ago