EKS AutoMode static PV can't be attached - volume attachment is being deleted

Question

I'm trying to migrating PV which manually bind to EBS to EKS auto mode, however after following below step suggested in [here](https://repost.aws/zh-Hans/questions/QUpT2qLrHwRDSiAGrIOhcP3w/eks-automode-node-can-t-bind-to-static-pv-does-not-contain-driver-ebs-csi-aws-com)

the pod it still shown that AttachVolume.Attach failed for volume "jenkins-pv" : volume attachment is being deleted

```
Events:
  Type     Reason              Age               From                     Message
  ----     ------              ----              ----                     -------
  Normal   Scheduled           12s               default-scheduler        Successfully assigned jenkins/jenkins-0 to i-08f1e11b490d30c9e
  Warning  FailedAttachVolume  1s (x9 over 11s)  attachdetach-controller  AttachVolume.Attach failed for volume "jenkins-pv" : volume attachment is being deleted
```

PV
```
Name:            jenkins-pv
Labels:          
Annotations:     pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    eks-auto
Status:          Bound
Claim:           jenkins/jenkins
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        30Gi
Node Affinity:   
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            ebs.csi.eks.amazonaws.com
    FSType:            ext4
    VolumeHandle:      vol-05a44cf51e421b306
    ReadOnly:          false
    VolumeAttributes:  
Events:                
```

PVC
```
me:          jenkins
Namespace:     jenkins
StorageClass:  eks-auto
Status:        Bound
Volume:        jenkins-pv
Labels:        app.kubernetes.io/component=jenkins-controller
               app.kubernetes.io/instance=jenkins
               app.kubernetes.io/managed-by=Helm
               app.kubernetes.io/name=jenkins
               helm.sh/chart=jenkins-5.8.15
Annotations:   meta.helm.sh/release-name: jenkins
               meta.helm.sh/release-namespace: jenkins
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      30Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       jenkins-0
Events:        
```

StorageClass
```
Name:            eks-auto
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"eks-auto"},"parameters":{"encrypted":"true","type":"gp3"},"provisioner":"ebs.csi.eks.amazonaws.com","volumeBindingMode":"Immediate"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           ebs.csi.eks.amazonaws.com
Parameters:            encrypted=true,type=gp3
AllowVolumeExpansion:  
MountOptions:          
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                
```

kevin · Accepted Answer

turns out i miss understanding the tag name
the key should be eks:eks-cluster-name instead of eks:our-custom-eks-name

re:Post Agent · Answer

The issue you're experiencing with your EKS AutoMode static PV not attaching properly is likely due to a conflict in the volume attachment process. Here are some steps to troubleshoot and resolve the issue:

1. First, ensure that the EBS volume is not attached to any other EC2 instance. If it is, detach it manually from the AWS console or using AWS CLI.

2. Double-check that the EBS volume has the correct tag. It should have a tag with the key 'eks:eks-cluster-name' and the value set to your cluster name.

3. Verify that your PV definition is using the correct CSI driver. The 'Driver' field in your PV definition looks correct (ebs.csi.eks.amazonaws.com), which is the appropriate driver for EKS AutoMode.

4. If the issue persists, try deleting the PV and PVC, then recreate them. This can help resolve any lingering attachment issues.

5. Ensure that the EBS volume is in the same Availability Zone as the node where the pod is scheduled. EBS volumes can only be attached to EC2 instances in the same AZ.

6. Check the EKS control plane logs for any errors related to volume attachment. You may need to enable control plane logging if it's not already enabled.

7. Verify that the IAM roles associated with your EKS nodes have the necessary permissions to attach EBS volumes.

If these steps don't resolve the issue, you may need to force the deletion of any stuck volume attachments using the AWS API or CLI. However, be cautious with this approach as it can potentially lead to data loss if not done correctly.

Remember, when migrating from manually bound EBS volumes to EKS AutoMode, you need to ensure that all the configurations are updated to work with the new provisioner and that there are no conflicts with existing attachments or outdated configurations.  
**Sources**  
[EKS AutoMode node can't bind to static PV - does not contain driver ebs.csi.aws.com | AWS re:Post](https://repost.aws/questions/QUpT2qLrHwRDSiAGrIOhcP3w/eks-automode-node-can-t-bind-to-static-pv-does-not-contain-driver-ebs-csi-aws-com)  
[Amazon EBS CSI Driver add-on compatibility with EKS compute resource Auto Mode configuration | AWS re:Post](https://repost.aws/questions/QUf8vA3FtVQwiTgGlcJwNWxA/amazon-ebs-csi-driver-add-on-compatibility-with-eks-compute-resource-auto-mode-configuration)

EKS AutoMode static PV can't be attached - volume attachment is being deleted

Add your answer

Relevant content