Secrets Store CSI Driver failing with Pod Identity on EKS Auto Mode

Question

Describe the bug

Secrets Store CSI Driver fails to retrieve secrets from AWS Secrets Manager when using EKS Pod Identity on EKS Auto Mode nodes. The driver is unable to obtain AWS credentials, causing the GetSecretValue API call to fail after exhausting all retry attempts.

Logs from Pod -

```
MountVolume.SetUp failed for volume "csi" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod , err: rpc error: code = Unknown desc = Failed to fetch secret from all regions. Verify secret exists and required permissions are granted for: 
```

Logs from aws driver for secrets -

```
operation error Secrets Manager: GetSecretValue, exceeded maximum number of attempts, 3,
get identity: get credentials: failed to refresh cached credentials,
failed to load credentials, exceeded maximum number of attempts, 3, : []
```

Provider class -

```
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  labels:
    argocd.argoproj.io/instance: secrets-provider-class-
  name: aws-secrets-provider-class
  namespace: <>
spec:
  parameters:
    objects: |
      - objectName: ""
        objectType: "secretsmanager"
        objectAlias: ""
    region: us-west-2
    usePodIdentity: 'true'
  provider: aws
```

AWS Secrets CSI Driver being used is from AWS EKS Add on with version - v3.0.0-eksbuild.1
AWS EKS Version - 1.3.4

Florian Turnwald · Answer

Since the issue only occurs with new pods while existing ones work, your IAM Trust Policy and permissions are likely correct. This behavior typically points to a **race condition** or **dependency delay** on newly provisioned Auto Mode nodes:
 * **Pod Identity Agent Readiness:** In EKS Auto Mode, the eks-pod-identity-agent must be fully operational on a new node before the CSI driver can retrieve credentials. If your application pod is scheduled and attempts to mount the secret before the agent is ready, the credential refresh will fail.
 * **DaemonSet Initialization:** Verify that the csi-secrets-store-provider-aws and the CSI driver pods have reached a Running state on the new node before the application pod starts.
 * **Recommendation:** Check the logs of the eks-pod-identity-agent pod on the specific node where the failure occurred. If you see failed 404 or 500 attempts around the time of the pod's creation, consider implementing a startup delay or checking node initialization sequences to ensure the Identity Agent has priority.

Riku_Kobayashi · Answer

Hello.

The EKS Pod Identity may not be functioning correctly, and the Secrets Store CSI Driver may not be able to retrieve the credentials.  
You may need to verify that the IAM role is configured correctly for the service account and that there are no issues with the IAM role's trust policy.  
https://docs.aws.amazon.com/eks/latest/userguide/pod-id-association.html

Secrets Store CSI Driver failing with Pod Identity on EKS Auto Mode

Add your answer

Relevant content