Skip to content

EKS AutoMode node can't bind to static PV - does not contain driver ebs.csi.aws.com

0

I'm trying to migrating PV to EKS auto mode, however after creating new storage class with provisioner ebs.csi.eks.amazonaws.com and re-create PV with new storage class, it seems that the node is still trying to use the old ebs.csi.aws.com driver, and I do add the tag eks:our-cluster-name our-cluster-name to the volume which is the only further action need to be taken that I found

from the pod

 Normal   Scheduled           20m                  default-scheduler        Successfully assigned jenkins/jenkins-0 to i-0c4100d0ac6a43cc3
 Warning  FailedAttachVolume  108s (x17 over 20m)  attachdetach-controller  AttachVolume.Attach failed for volume "jenkins-pv" : CSINode i-0c4100d0ac6a43cc3 does not contain driver ebs.csi.aws.com

storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: eks-auto
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.eks.amazonaws.com
volumeBindingMode: Immediate
parameters:
  type: gp3
  encrypted: "true"

PV

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
  namespace: jenkins
spec:
  storageClassName: eks-auto
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 30Gi
  persistentVolumeReclaimPolicy: Retain
  awsElasticBlockStore:
    volumeID: vol-05a44cf51e421b306
    fsType: ext4

PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins
  namespace: jenkins
  uid: e56b7001-ad97-495e-bbf5-bf5f1160e249
  resourceVersion: '94066643'
  creationTimestamp: '2025-02-19T09:57:58Z'
  labels:
    app.kubernetes.io/component: jenkins-controller
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-5.8.14
  annotations:
    meta.helm.sh/release-name: jenkins
    meta.helm.sh/release-namespace: jenkins
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
  finalizers:
    - kubernetes.io/pvc-protection
  selfLink: /api/v1/namespaces/jenkins/persistentvolumeclaims/jenkins
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 30Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  volumeName: jenkins-pv
  storageClassName: eks-auto
  volumeMode: Filesystem
1 Answer
1
Accepted Answer

The issue you're experiencing is related to the migration of your persistent volumes (PVs) to EKS Auto Mode. There are a few key points to consider:

  1. EKS Auto Mode uses a different provisioner for EBS volumes. The correct provisioner is 'ebs.csi.eks.amazonaws.com', which you've correctly set in your StorageClass.

  2. For static provisioning with existing EBS volumes in EKS Auto Mode, you need to add the AWS tag 'eks:eks-cluster-name' with the value of your cluster name to the volume. You've mentioned doing this, which is correct.

  3. The error message suggests that the node is still looking for the old EBS CSI driver ('ebs.csi.aws.com'). This indicates that the migration process might not be complete or there's a configuration issue.

  4. In your PV definition, you're still using the 'awsElasticBlockStore' volume type. For EKS Auto Mode, you should use the CSI volume type instead.

To resolve this issue:

  1. Update your PV definition to use the CSI volume type. Replace the 'awsElasticBlockStore' section with:
csi:
  driver: ebs.csi.eks.amazonaws.com
  volumeHandle: vol-05a44cf51e421b306
  fsType: ext4
  1. Ensure that all nodes in your cluster are updated to support EKS Auto Mode and the new EBS CSI driver.

  2. Verify that the EBS CSI driver add-on is not installed on your cluster, as it's not needed for EKS Auto Mode.

  3. Double-check that the AWS tag on your EBS volume is correct and matches your cluster name exactly.

  4. If the issue persists, you may need to delete and recreate the PV and PVC to ensure they're properly configured for EKS Auto Mode.

Remember that when transitioning to EKS Auto Mode, you need to plan for migrating your volumes carefully. This process may involve creating volume snapshots, updating volume tags, and modifying your PV and PVC definitions to work with the new provisioner.
Sources
Amazon EBS CSI Driver add-on compatibility with EKS compute resource Auto Mode configuration | AWS re:Post
Store Kubernetes volumes with Amazon EBS - Amazon EKS
Create a storage class - Amazon EKS

answered 2 years ago

EXPERT

reviewed 2 years ago

  • I've followed above action step by step, I could see the PVC bound to PV but on the pod it still shown that AttachVolume.Attach failed for volume "jenkins-pv" : volume attachment is being deleted

    Events:
      Type     Reason              Age               From                     Message
      ----     ------              ----              ----                     -------
      Normal   Scheduled           12s               default-scheduler        Successfully assigned jenkins/jenkins-0 to i-08f1e11b490d30c9e
      Warning  FailedAttachVolume  1s (x9 over 11s)  attachdetach-controller  AttachVolume.Attach failed for volume "jenkins-pv" : volume attachment is being deleted
    

    PV

    Name:            jenkins-pv
    Labels:          <none>
    Annotations:     pv.kubernetes.io/bound-by-controller: yes
    Finalizers:      [kubernetes.io/pv-protection]
    StorageClass:    eks-auto
    Status:          Bound
    Claim:           jenkins/jenkins
    Reclaim Policy:  Retain
    Access Modes:    RWO
    VolumeMode:      Filesystem
    Capacity:        30Gi
    Node Affinity:   <none>
    Message:
    Source:
        Type:              CSI (a Container Storage Interface (CSI) volume source)
        Driver:            ebs.csi.eks.amazonaws.com
        FSType:            ext4
        VolumeHandle:      vol-05a44cf51e421b306
        ReadOnly:          false
        VolumeAttributes:  <none>
    Events:                <none>
    

    PVC

    Name:          jenkins
    Namespace:     jenkins
    StorageClass:  eks-auto
    Status:        Bound
    Volume:        jenkins-pv
    Labels:        a
    

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.