Why are my Pods stuck in the Pending state after I switched from standard Amazon EKS nodes to EKS Auto Mode?

Resolution

Determine why the Pods are stuck in the Pending state

Run the following command on the Pods that are stuck in the Pending state:

kubectl describe pods

If you receive the following error message in the output, then you must create a new storage class:

"Warning FailedScheduling 13s eks-auto-mode/compute Failed to schedule pod, failed to validate pvc, provisioner is not supported (PersistentVolumeClaim=default/ebs-claim, Provisioner=ebs.csi.aws.com, StorageClass=ebs-sc)"

If you receive the following error message in the output, then you must replace your current storage class:

"Warning FailedAttachVolume 36s (x9 over 2m48s) attachdetach-controller AttachVolume.Attach failed for volume "volumeID1234" : CSINode i-08222xyz does not contain driver ebs.csi.aws.com"

Important: When you migrate from standard Amazon EKS nodes to EKS Auto Mode managed nodes, you must modify the storage class configuration. Change the Amazon Elastic Block Store (Amazon EBS) CSI driver provisioner to the EKS Auto Mode CSI provisioner.

Create a new storage class

To create a new storage class for the EKS Auto Mode provisioner that runs in parallel with your current storage class, complete the following steps:

1. Use the following template to create a StorageClass Kubernetes manifest and name it eks-auto-mode-sc.yaml:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: auto-ebs-sc
   allowedTopologies:
   - matchLabelExpressions:
     - key: eks.amazonaws.com/compute-type
       values:
       - auto
   provisioner: ebs.csi.eks.amazonaws.com
   volumeBindingMode: WaitForFirstConsumer
   parameters:
     type: gp3
     encrypted: "true"

   The template creates a storage class that's called auto-ebs-sc that runs only on EKS Auto Mode nodes and uses the allowedTopologies key.

2. To install the new storage class, run the following command:

   kubectl apply -f eks-auto-mode-sc.yaml

3. To update your workloads to use the new storage class, update all PersistentVolumeClaims (PVCs) in your workload manifests:

    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: auto-ebs-sc

   Note: The update doesn't automatically migrate existing persistent volumes.

4. To apply the manifest changes and migrate existing persistent volumes, relaunch your deployments and StatefulSets. For more information, see StatefulSets on the Kubernetes website.

Replace your current storage class

Complete the following steps:

1. To export your current storage class configuration, run the following commands:

   kubectl get sc SC-NAME -o yaml > YOUR_ORIGINAL_SC_CONFIG.yaml
   kubectl get sc SC-NAME -o yaml > YOUR_NEW_SC_CONFIG.yaml

2. In the YOUR_NEW_SC_CONFIG.yaml file, replace the ebs.csi.aws.com provisioner with the ebs.csi.eks.amazonaws.com EKS Auto Mode provisioner.
   Note: If your storage class has other customizations, then review them against the EKS Auto Mode parameters and requirements before you complete the next step.

3. To delete the existing storage class, run the following command:

   kubectl delete sc YOUR_STORAGE_CLASS

   Note: Replace YOUR_STORAGE_CLASS with your storage class name.
   Example output:

   storageclass.storage.k8s.io "ebs-sc" deleted

4. To apply the new configuration's YAML file, run the following command:

   kubectl apply -f YOUR_NEW_SC_CONFIG.yaml

   Note: Replace YOUR_NEW_SC_CONFIG with the name the configuration file for your new storage class.
   Example output:

   storageclass.storage.k8s.io/ebs-sc created

5. To update your workloads to use the new storage class, update all PVCs in your workload manifests:

    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: auto-ebs-sc

   Note: The update doesn't automatically migrate existing persistent volumes.

6. To apply the manifest changes and migrate existing persistent volumes, relaunch your deployments and StatefulSets.

Reconfigure persistent volumes for migration to the EKS Auto Mode EBS CSI storage class

Use the eks-auto-mode-ebs-migration-tool to migrate the standard EBS CSI storage class ebs.csi.aws.com to the EKS Auto EBS CSI storage class ebs.csi.eks.amazonaws.com. The migration reconfigures the persistent volumes to use the new storage class configuration and attaches the workloads to the configuration. To download the tool, see eks-auto-mode-ebs-migration-tool on the GitHub website.