How do I use persistent storage in Amazon EKS?

Short description

To set up persistent storage in Amazon EKS, use either of the following options:

• Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver
• Amazon Elastic File System (Amazon EFS) Container Storage Interface (CSI) driver

To use one of these options, complete the steps in either of the following sections:

• Option A: Deploy and test the Amazon EBS CSI driver
• Option B: Deploy and test the Amazon EFS CSI driver

The commands in this article require kubectl version 1.14 or later. To see your version of kubectl, run the following command:

kubectl version --client --short

Note: It's a best practice to make sure you install the latest version of the drivers. For more information, see the GitHub repositories for the Amazon EBS CSI driver and Amazon EFS CSI driver.

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you're using the most recent version of the AWS CLI.

Before you complete the steps in either section, you must:

1. Install the AWS CLI.

2. Set AWS Identity and Access Management (IAM) permissions to create and attach a policy to the Amazon EKS worker node role CSI Driver Role.
   Note: The commands in the following sections don't always use the --region variable. If you don't use --region, then the commands use the default value for your AWS Region. To check the default value, run the AWS CLI configure command.

3. Create your Amazon EKS cluster and join your worker nodes to the cluster.
   Note: Run the kubectl get nodes command to verify your worker nodes are attached to your cluster.

4. To verify that your AWS IAM OpenID Connect (OIDC) provider exists for your cluster, run the following command:

   aws eks describe-cluster --name your_cluster_name --query "cluster.identity.oidc.issuer" --output text

   Note: Replace your_cluster_name with your cluster name.

5. Verify that your IAM OIDC provider is configured:

   aws iam list-open-id-connect-providers | grep OIDC_PROVIDER_ID

   Note: Replace ID of the oidc provider with your OIDC ID. If you receive a No OpenIDConnect provider found in your account error, you must create an IAM OIDC provider.

6. Install or update eksctl.

7. Create an IAM OIDC provider:

   eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve 

   Note: Replace my-cluster with your cluster name.

Option A: Deploy and test the Amazon EBS CSI driver

Deploy the Amazon EBS CSI driver:

1. Create an IAM trust policy file, similar to the following example:

   cat <<EOF > trust-policy.json
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID:oidc-provider/oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/YOUR_OIDC ID"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/<XXXXXXXXXX45D83924220DC4815XXXXX>:aud": "sts.amazonaws.com",
             "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/<XXXXXXXXXX45D83924220DC4815XXXXX>:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
           }
         }
       }
     ]
   }
   EOF

   Note: Replace YOUR_AWS_ACCOUNT_ID with your account ID. Replace YOUR_AWS_REGION with your Region. Replace your OIDC ID with the output from creating your IAM OIDC provider.

2. Create an IAM role named Amazon_EBS_CSI_Driver:

   aws iam create-role \
    --role-name AmazonEKS_EBS_CSI_Driver \
    --assume-role-policy-document file://"trust-policy.json"

3. Attach the AWS managed IAM policy for the EBS CSI Driver to the IAM role that you created:

   aws iam attach-role-policy \
   --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
   --role-name AmazonEKS_EBS_CSI_Driver

4. Deploy the Amazon EBS CSI driver.

   aws eks create-addon \
    --cluster-name my-cluster \
    --addon-name aws-ebs-csi-driver \
    --service-account-role-arn arn:aws:iam::
   YOUR_AWS_ACCOUNT_ID:role/AmazonEKS_EBS_CSI_DriverRole

   Note: Replace my-cluster with your cluster name and YOUR_AWS_ACCOUNT_ID with your account ID.

   You can deploy the EBS CSI driver with Kustomize, Helm, or an Amazon EKS managed add-on. This example uses the Amazon EKS add-on feature to deploy the driver. For more information, see the aws-ebs-csi-driver installation guide on GitHub.

5. Check that the EBS CSI driver installed successfully:

   eksctl get addon --cluster my-cluster | grep ebs

   A successfully installation returns the following output:

   aws-ebs-csi-driver    v1.20.0-eksbuild.1    ACTIVE    0    arn:aws:iam::YOUR_AWS_ACCOUNT_ID:role/AmazonEKS_EBS_CSI_Driver

Test the Amazon EBS CSI driver:

You can test your Amazon EBS CSI driver with a sample application that uses dynamic provisioning for the pods. The Amazon EBS volume is provisioned on demand.

1. Clone the aws-ebs-csi-driver repository from AWS GitHub:

   git clone https://github.com/kubernetes-sigs/aws-ebs-csi-driver.git

2. Change your working directory to the folder that contains the Amazon EBS driver test files:

   cd aws-ebs-csi-driver/examples/kubernetes/dynamic-provisioning/

3. Create the Kubernetes resources required for testing:

   kubectl apply -f manifests/

   Note: The kubectl command creates a StorageClass, PersistentVolumeClaim (PVC), and pod. The pod references the PVC. An Amazon EBS volume is provisioned only when the pod is created. For more information, see Storage classes and PersistentVolumeClaims on the Kubernetes website.

4. Describe the ebs-sc storage class:

   kubectl describe storageclass ebs-sc

5. Watch the pods in the default namespace and wait for the app pod's status to change to Running. For example:

   kubectl get pods --watch

6. View the persistent volume created because of the pod that references the PVC:

   kubectl get pv

7. View information about the persistent volume:

   kubectl describe pv your_pv_name

   Note: Replace your_pv_name with the name of the persistent volume returned from step 6 in the previous section. The value of the Source.VolumeHandle property in the output is the ID of the physical Amazon EBS volume created in your account.

8. Verify that the pod is writing data to the volume:

   kubectl exec -it app -- cat /data/out.txt

   Note: The command output displays the current date and time stored in the /data/out.txt file. The file includes the day, month, date, and time.

Option B: Deploy and test the Amazon EFS CSI driver

Before you deploy the CSI driver, create an IAM role that allows the CSI driver's service account to make calls to AWS APIs on your behalf.

1. Download the IAM policy document from GitHub:

   curl -o iam-policy-example.json https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/docs/iam-policy-example.json

2. Create an IAM policy:

   aws iam create-policy \
       --policy-name AmazonEKS_EFS_CSI_Driver_Policy \
       --policy-document file://iam-policy-example.json

3. Run the following command to determine your cluster's OIDC provider URL:

   aws eks describe-cluster --name your_cluster_name --query "cluster.identity.oidc.issuer" --output text

   Note: In step 3, replace your_cluster_name with your cluster name.

4. Create the following IAM trust policy, and then grant the AssumeRoleWithWebIdentity action to your Kubernetes service account:

   cat <<EOF > trust-policy.json
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID:oidc-provider/oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/<XXXXXXXXXX45D83924220DC4815XXXXX>"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/<XXXXXXXXXX45D83924220DC4815XXXXX>:sub": "system:serviceaccount:kube-system:efs-csi-controller-sa"
           }
         }
       }
     ]
   }
   EOF

   Note: In step 4, replace YOUR_AWS_ACCOUNT_ID with your account ID. Replace YOUR_AWS_REGION with your Region. Replace XXXXXXXXXX45D83924220DC4815XXXXX with the value returned in step 3.

5. Create an IAM role:

   aws iam create-role \
     --role-name AmazonEKS_EFS_CSI_DriverRole \
     --assume-role-policy-document file://"trust-policy.json"

6. Attach your new IAM policy to the role:

   aws iam attach-role-policy \
     --policy-arn arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AmazonEKS_EFS_CSI_Driver_Policy \
     --role-name AmazonEKS_EFS_CSI_DriverRole

7. Save the following contents to a file named efs-service-account.yaml.

   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     labels:
       app.kubernetes.io/name: aws-efs-csi-driver
     name: efs-csi-controller-sa
     namespace: kube-system
     annotations:
       eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/AmazonEKS_EFS_CSI_DriverRole

8. Create the Kubernetes service account on your cluster. The Kubernetes service account named efs-csi-controller-sa is annotated with the IAM role that you created.

   kubectl apply -f efs-service-account.yaml

9. Install the driver using images stored in the public Amazon ECR registry by downloading the manifest:

   kubectl kustomize "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.5" > public-ecr-driver.yaml

   Note: You can install the EFS CSI Driver using Helm and a Kustomize with AWS Private or Public Registry. For more information, see the Amazon EFS CSI Driver documentation.

   Then, edit the public-ecr-driver.yaml file to remove the efs-csi-controller-sa manifest:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     labels:
       app.kubernetes.io/name: aws-efs-csi-driver
     name: efs-csi-controller-sa
     namespace: kube-system

Deploy the Amazon EFS CSI driver

The Amazon EFS CSI driver allows multiple pods to write to a volume at the same time with the ReadWriteMany mode.

1. To deploy the Amazon EFS CSI driver, apply the manifest:

   kubectl apply -f public-ecr-driver.yaml

2. If your cluster contains only AWS Fargate pods (no nodes), then deploy the driver with the following command (all Regions):

   kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/deploy/kubernetes/base/csidriver.yaml

Create an Amazon EFS File system

1. Get the VPC ID for your Amazon EKS cluster:

   aws eks describe-cluster --name your_cluster_name --query "cluster.resourcesVpcConfig.vpcId" --output text

   Note: In step 3, replace your_cluster_name with your cluster name.

2. Get the CIDR range for your VPC cluster:

   aws ec2 describe-vpcs --vpc-ids YOUR_VPC_ID --query "Vpcs[].CidrBlock" --output text

   Note: In step 4, replace the YOUR_VPC_ID with the VPC ID from step 3 in the previous section.

3. Create a security group that allows inbound network file system (NFS) traffic for your Amazon EFS mount points:

   aws ec2 create-security-group --description efs-test-sg --group-name efs-sg --vpc-id YOUR_VPC_ID

   Note: Replace YOUR_VPC_ID with the output from step 3 in the previous section. Save the GroupId for later.

4. Add an NFS inbound rule so that resources in your VPC can communicate with your Amazon EFS file system:

   aws ec2 authorize-security-group-ingress --group-id sg-xxx --protocol tcp --port 2049 --cidr YOUR_VPC_CIDR

   Note: Replace YOUR_VPC_CIDR with the output from step 4 in the previous section. Replace sg-xxx with the security group ID from step 5 in the previous section.

5. Create an Amazon EFS file system for your Amazon EKS cluster:

   aws efs create-file-system --creation-token eks-efs

   Note: Save the FileSystemId for later use.

6. To create a mount target for Amazon EFS, run the following command:

   aws efs create-mount-target --file-system-id FileSystemId --subnet-id SubnetID --security-group sg-xxx

   Important: Be sure to run the command for all the Availability Zones with the SubnetID in the Availability Zone where your worker nodes are running. Replace FileSystemId with the output of step 7 in the previous section (where you created the Amazon EFS file system). Replace sg-xxx with the output of the preceding step 5 (where you created the security group). Replace SubnetID with the subnet used by your worker nodes. To create mount targets in multiple subnets, you must run the command in step 8 separately for each subnet ID. It's a best practice to create a mount target in each Availability Zone where your worker nodes are running.

   Note: You can create mount targets for all the Availability Zones where worker nodes are launched. Then, all the Amazon Elastic Compute Cloud (Amazon EC2) instances in the Availability Zone with the mount target can use the file system.

The Amazon EFS file system and its mount targets are now running and ready to be used by pods in the cluster.

Test the Amazon EFS CSI driver

You can test the Amazon EFS CSI driver by deploying two pods that write to the same file.

1. Clone the aws-efs-csi-driver repository from AWS GitHub:

   git clone https://github.com/kubernetes-sigs/aws-efs-csi-driver.git

2. Change your working directory to the folder that contains the Amazon EFS CSI driver test files:

   cd aws-efs-csi-driver/examples/kubernetes/multiple_pods/

3. Retrieve your Amazon EFS file system ID that was created earlier:

   aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text

   Note: If the command in step 3 returns more than one result, then you can use the Amazon EFS file system ID that you saved earlier.

4. In the specs/pv.yaml file, replace the spec.csi.volumeHandle value with your Amazon EFS FileSystemId from previous steps.

5. Create the Kubernetes resources required for testing:

   kubectl apply -f specs/

   Note: The kubectl command in the preceding step 5 creates an Amazon EFS storage class, PVC, persistent volume, and two pods (app1 and app2).

6. List the persistent volumes in the default namespace, and look for a persistent volume with the default/efs-claim claim:

   kubectl get pv efs-pv

7. Describe the persistent volume:

   kubectl describe pv efs-pv

8. Test if the two pods are writing data to the file:

   kubectl exec -it app1 -- tail /data/out1.txt  
   kubectl exec -it app2 -- tail /data/out1.txt

   Wait for about one minute. The output shows the current date written to /data/out1.txt by both pods.

   Related information

   Troubleshooting Amazon EFS