EKS and IAM OIDC provider and Token Expiration



Brand new EKS cluster latest version. Followed the first example in this guide: https://docs.aws.amazon.com/eks/latest/userguide/cross-account-access.html

Created an OIDC Identity provider on Account1 accepting requests from the EKS cluster on account 2. In the EKS cluster, my k8s ServiceAccount resource have an annotation eks.amazonaws.com/role-arn pointing to an IAM role in account1. Application running in the pod is a .NET6 app with the AWSSDK.DynamoDBv2 nuget package making DynamoDB queries.

It worked for a while, until at some point I got this exception:

Amazon.Runtime.AmazonClientException: Error calling AssumeRole for role arn:aws:iam::AcccountNumber:role/EKS-ServiceAccount
 ---> Amazon.SecurityToken.Model.ExpiredTokenException: Token expired: current date/time 1680295159 must be before the expiration date/time1680281898
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.

I do see doing a kubectl describe on my pod these information:

      AWS_STS_REGIONAL_ENDPOINTS:        regional
      AWS_DEFAULT_REGION:                us-east-1
      AWS_REGION:                        us-east-1
      AWS_ROLE_ARN:                      arn:aws:iam::AcccountNumber:role/EKS-ServiceAccount
      AWS_WEB_IDENTITY_TOKEN_FILE:       /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mq27b (ro)
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400

I also found this page mentioning it should renew at 80% expiration time and this page with the minimum required SDK version. I can confirm I use AWSSDK.DynamoDBv2, AWSSDK.SecurityToken and AWSSDK.Core all version later than that (

I was expecting the EKS cluster to automatically renew the token from the OIDC provider. Why isn't it doing it?

1 Answer

Service account tokens have an expiration of one hour. In earlier Kubernetes versions, the tokens didn't have an expiration. This means that clients that rely on these tokens must refresh the tokens within an hour. The following Kubernetes client SDKs refresh tokens automatically within the required time frame:

  • Go version 0.15.7 and later

  • Python version 12.0.0 and later

  • Java version 9.0.0 and later

  • JavaScript version 0.10.3 and later

  • Ruby master branch

  • Haskell version

  • C# version 7.0.5 and later

If your workload is using an earlier client version, then you must update it. [1]

The application is responsible for reloading the token when it rotates. It's often good enough for the application to load the token on a schedule (for example: once every 5 minutes), without tracking the actual expiry time.[2]

What is the EKS version are you running?

[1] https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#service-account-tokens

[2] https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

profile picture
answered a year ago
  • Hello @bijay_k, thanks for the reply

    My EKS cluster version is 1.25 My pods have been redeployed 26hours ago and queries still seems to work, so I'm not sure if the problem was related due to something else. I'll keep this topic updated with more information if the problem arise again.

    I'm not sure I follow, you are suggesting all container applications requires to reference the Kubernetes client SDKs and refresh the token manually inside a worker thread, and the AWS SDK doesn't do it on its own? This would be extremely tedious and it wouldn't work with most public docker applications if it requires custom code. To clarify, the application doesn't interact with the cluster, it only uses services via the AWS SDK, which I would expect to automatically refresh its connection information via the refreshed token sent via the projected volume.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions