How do I troubleshoot issues with my Amazon EBS volume mounts in Amazon EKS?

Resolution

Prerequisites:

• Set the required AWS Identity and Access Management (IAM) permissions for the ebs-csi-controller-sa service account in your IAM policy.
• Verify that you have a valid PersistentVolumeClaim (PVC) in the same namespace as your Pod. For an example PVC file, see claim.yaml on the GitHub website.
• Make sure that you have a valid Amazon EBS storage class definition that uses the Amazon EBS Container Storage Interface (CSI) driver provisioner ebs.csi.aws.com. For more information, see Amazon EBS on the Kubernetes website.

Verify that the Amazon EBS CSI driver controller and node Pods are running

The Amazon EBS CSI driver consists of controller Pods that run as a deployment and node Pods that run as a daemon set. To verify that your cluster runs these Pods, run the following command:

kubectl get all -l app.kubernetes.io/name=aws-ebs-csi-driver -n kube-system

Note: Windows worker nodes and AWS Fargate don't support the Amazon EBS CSI driver.

Make sure that the installed Amazon EBS CSI driver version is compatible with your cluster's Kubernetes version. For more information, see Amazon EBS CSI driver on the GitHub website.

If your cluster doesn't run these Pods, then get the Amazon EBS CSI driver.

Check for PVC issues

To check whether the PVC encounters issues, run the following command to view events:

kubectl describe pvc PVC_NAME -n NAMESPACE

Note: Replace PVC_NAME with your PVC's name and NAMESPACE with your namespace.

If you use dynamic volume provisioning, then review the returned output of events to determine whether volume provisioning succeeded or failed. You can also see the persistent volume name that's bound to the PVC.

Example output:

Name:          ebs-claimNamespace:     default
StorageClass:  ebs-sc
Status:        Bound
Volume:        pvc-12345678-1234-1234-1234-123456789abc
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-10-0-2-57.ec2.internal
. . . . .
. . . . . 
Events:
  Type    Reason                 Age                    From                                                                                      Message
  ----    ------                 ----                   ----                                                                                      -------
. . . . .
  Normal  Provisioning           5m22s                  ebs.csi.aws.com_ebs-csi-controller-5a6b7c8d9e-f1g2h_12345678-1234-1234-1234-123456789abc  External provisioner is provisioning volume for claim "default/ebs-claim"
  Normal  ProvisioningSucceeded  5m18s                  ebs.csi.aws.com_ebs-csi-controller-5a6b7c8d9e-f1g2h_12345678-1234-1234-1234-123456789abc  Successfully provisioned volume pvc-12345678-1234-1234-1234-123456789abc

Review the logs from the Amazon EBS CSI controller Pods

If the volume fails during creation, then refer to the ebs-plugin and csi-provisioner logs.

To retrieve the ebs-plugin container log, run the following commands:

kubectl logs deployment/ebs-csi-controller -n kube-system -c ebs-plugin
kubectl logs daemonset/ebs-csi-node -n kube-system -c ebs-plugin

To retrieve the csi-provisioner container logs, run the following command:

kubectl logs deployment/ebs-csi-controller -n kube-system -c csi-provisioner

If the Amazon EBS volumes fail to attach to the Pod, then review the csi-attacher logs. To retrieve the csi-attacher container logs, run the following command:

kubectl logs deployment/ebs-csi-controller -n kube-system -c csi-attacher

Look for information in the logs that can help you troubleshoot the error. For more information, see AWSSupport-TroubleshootEbsCsiDriversForEks.

Verify IAM permissions for the Amazon EBS CSI controller

Note: To increase the debugging efficiency of the Amazon EBS CSI driver, set aws-sdk-debug-log to true. For more information, see Driver options on the GitHub website.

The following issues result in unauthorized errors in your PVC events or in your ebs-csi-controller log:

• Missing permissions in the IAM role
• An incorrect IAM role that's associated with the service account

To resolve these issues, annotate the Amazon EBS CSI driver controller service account with the correct IAM role and permissions. To check the service account's IAM role and permissions, complete the following steps:

1. To determine if the ebs-csi-controller Pods' service account has the correct annotation, run the following command:

   kubectl get sa ebs-csi-controller-sa -n kube-system -o jsonpath='{.metadata.annotations}'

   Note: IAM roles for service accounts (IRSA) use the annotation to link the IAM role to the service account.

2. In the command output, verify that the following annotation is present:

   eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/AmazonEKS_EBS_CSI_DriverRole

3. Verify that you created the IAM OpenID Connect (OIDC) provider for your cluster.

4. Verify that the IAM role has permissions to perform Amazon EBS API calls. Make sure that the IAM role's trust policy trusts the service account ebs-csi-controller-sa.

5. Verify that AWS CloudTrail makes the calls for CreateVolume, AttachVolume, and DetachVolume. To determine whether the controller's service account IAM role or the worker node's IAM role is making the API calls, review the CloudTrail logs. The logs show which IAM identity is performing the actions.

Resolve volume scheduling and attachment errors

Each persistent volume has a node affinity that limits the attachment of persistent volumes to nodes within a single Availability Zone. This is because you can attach Amazon EBS volumes only to Pods or nodes that run in the same Availability Zone that you created them in. If the scheduled Pods for nodes in one Availability Zone use the persistent volume for Amazon EBS in a different Availability Zone, then you receive this error:

"FailedScheduling: 1 node(s) had volume node affinity conflict"

To troubleshoot this error, use StatefulSets instead of a deployment. This creates a unique Amazon EBS volume for each Pod of the StatefulSets in the same Availability Zone as the Pod. For more information, see StatefuleSets on the Kubernetes website.

Important: You can't mount an Amazon EBS volume to two different Pods that run on two different worker nodes. You can attach the Amazon EBS volume to Pods that run on one node, but you can't attach it to another node at the same time.

To verify the persistent volume's node affinity, run the following command:

kubectl describe pv PERSISTENT_VOLUME_NAME

Note: Replace PERSISTENT_VOLUME_NAME with your persistent volume's name.

If you try to attach your Amazon EBS volume to two Pods on different worker nodes and the Pod fails, then you might get the following error:

"Warning FailedAttachVolume 2m38s attachdetach-controller Multi-Attach error for volume "pvc-1cccsdfdc8-fsdf6-43d6-a1a9-ea837hf7h57fa" Volume is already exclusively attached to one node and can't be attached to another"

To resolve this error, verify that only one Pod at a time uses a persistent volume backed by Amazon EBS on a single node. Amazon EBS volumes support only the ReadWriteOnce (RWO) access mode. This means that you can attach the volume to one node at a time only.

Verify that your Amazon EBS controller Pods can connect to the Amazon EC2 API

If you have connection timeout errors in the ebs-csi-controller logs, then the Amazon EBS CSI controller might not be connected to the Amazon Elastic Compute Cloud (Amazon EC2) API. If the controller Pods have connectivity issues when you create your PVC, then you might get the following error:

"Warning ProvisioningFailed persistentvolumeclaim/storage-volume-1 failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded"

To resolve this error, verify that the subnets of the Amazon EBS controller Pods can connect to the Amazon EC2 API. If you run a private cluster with an HTTP/HTTPS proxy, then verify that your Amazon EBS CSI controller Pods can use the HTTP/HTTPS proxy. The Amazon EBS CSI driver's Helm installation supports setup of an HTTP/HTTPS proxy. For more information, see values.yaml on the GitHub website.

The preceding error message might also relate to an issue with OIDC. To trouble OIDC issues, see How do I troubleshoot an OIDC provider and IRSA in Amazon EKS?

Activate Amazon EBS encryption

Make sure that parameters.encrypted is set to true in the storage class manifest. To check the setting, run the following command:

kubectl describe sc STORAGE_CLASS_NAME

Note: Replace STORAGE_CLASS_NAME with your storage class name.

You can keep the AWS managed key for AWS Key Management Service (AWS KMS) with the alias aws/ebs created on your behalf as the default encryption key. Or, you can choose a symmetric customer managed encryption key. For more information, see Enable Amazon EBS encryption by default.

The Amazon EBS CSI driver requires access to customer managed keys that encrypt or decrypt the volume. Otherwise, volume creation times out and you can't create or manage encrypted Amazon EBS volumes.

To resolve this issue, make sure that the IAM role of the Amazon EBS CSI controller has the permissions from the following example:

{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
          ],
          "Resource": ["custom-key-arn"],
          "Condition": {
            "Bool": {
              "kms:GrantIsForAWSResource": "true"
            }
          }
        },
        {
          "Effect": "Allow",
          "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
          ],
          "Resource": ["custom-key-arn"]
        }
      ]
}

Set the AWS KMS key policy to explicitly allow the IAM role of the Amazon EBS CSI controller to access the AWS KMS key. For more information, see Required AWS KMS key policy for use with encrypted volumes.