EKS and IAM OIDC provider and Token Expiration

Question

Hello,

Brand new EKS cluster latest version.
Followed the first example in this guide: https://docs.aws.amazon.com/eks/latest/userguide/cross-account-access.html

Created an OIDC Identity provider on Account1 accepting requests from the EKS cluster on account 2.
In the EKS cluster, my k8s ServiceAccount resource have an annotation eks.amazonaws.com/role-arn pointing to an IAM role in account1.
Application running in the pod is a .NET6 app with the AWSSDK.DynamoDBv2 nuget package making DynamoDB queries.

It worked for a while, until at some point I got this exception:
```
Amazon.Runtime.AmazonClientException: Error calling AssumeRole for role arn:aws:iam::AcccountNumber:role/EKS-ServiceAccount
 ---> Amazon.SecurityToken.Model.ExpiredTokenException: Token expired: current date/time 1680295159 must be before the expiration date/time1680281898
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
```

I do see doing a kubectl describe on my pod these information:
```
    Environment:
      AWS_ACCESS_KEY_ID:
      AWS_SECRET_KEY:
      AWS_STS_REGIONAL_ENDPOINTS:        regional
      AWS_DEFAULT_REGION:                us-east-1
      AWS_REGION:                        us-east-1
      AWS_ROLE_ARN:                      arn:aws:iam::AcccountNumber:role/EKS-ServiceAccount
      AWS_WEB_IDENTITY_TOKEN_FILE:       /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mq27b (ro)
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
```

I also found [this page](https://docs.aws.amazon.com/eks/latest/userguide/pod-configuration.html) mentioning it should renew at 80% expiration time and [this page](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html) with the minimum required SDK version. I can confirm I use AWSSDK.DynamoDBv2, AWSSDK.SecurityToken and AWSSDK.Core all version later than that (3.7.100.14).

I was expecting the EKS cluster to automatically renew the token from the OIDC provider.
Why isn't it doing it?

bijay_k · Answer

Service account tokens have an expiration of one hour. In earlier Kubernetes versions, the tokens didn't have an expiration. This means that clients that rely on these tokens must refresh the tokens within an hour. The following Kubernetes client SDKs refresh tokens automatically within the required time frame:

* Go version 0.15.7 and later
 
* Python version 12.0.0 and later
 
* Java version 9.0.0 and later
 
* JavaScript version 0.10.3 and later
 
* Ruby master branch
 
* Haskell version 0.3.0.0
 
* C# version 7.0.5 and later
 
If your workload is using an earlier client version, then you must update it. [1]

The application is responsible for reloading the token when it rotates. It's often good enough for the application to load the token on a schedule (for example: once every 5 minutes), without tracking the actual expiry time.[2]

What is the EKS version are you running?

[1] https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#service-account-tokens

[2] https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

EKS and IAM OIDC provider and Token Expiration

Relevant content