AWS re:Post Knowledge Center Feedback Survey

Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.

How do I use persistent storage in Amazon EKS?

8 minute read

I want to use persistent storage in Amazon Elastic Kubernetes Service (Amazon EKS).

Short description

To use persistent storage in Amazon EKS, complete the steps for one of the following options:

Deploy and test the Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver.
Deploy and test the Amazon Elastic File System (Amazon EFS) CSI driver.

Note: For security and efficiency, it's a best practice to install the CSI driver through an Amazon EKS add-on. It's also a best practice to install the latest version of the drivers. For installation steps and the latest driver version, see aws-ebs-csi-driver or aws-efs-csi-driver on the GitHub website.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Prerequisites:

Set up a kubectl version that aligns with your Kubernetes version. For information about version skew requirements, see kubectl on the Kubernetes website.
Install or update eksctl. For instructions, see Installation on the eksctl website.
Set up AWS Identity and Access Management (IAM) Create permissions. Then, attach the policy to the CSI driver role or to the Amazon EKS worker node role.
Create your Amazon EKS cluster, and then join your worker nodes to the cluster. To verify that you attached your worker nodes to your cluster, run the following command:
```
kubectl get nodes
```

To verify that an IAM OpenID Connect (OIDC) provider exists for your cluster, run the following describe-cluster AWS CLI command:

aws eks describe-cluster
 --name your_cluster_name
 --query "cluster.identity.oidc.issuer"
 --output text

Note: Replace your_cluster_name with your cluster name.

To verify that you configured your IAM OIDC provider, run the following list-open-id-connect-providers command:
```
aws iam list-open-id-connect-providers | grep OIDC-ID
```
Note: Replace OIDC-ID with your OIDC provider ID. If you receive a "No OpenIDConnect provider found in your account" error message, then create an IAM OIDC provider.

Amazon EBS CSI driver

To deploy the Amazon EBS CSI driver, complete the following steps:

Create an IAM trust policy file that's similar to the following example:

cat <<EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID:oidc-provider/oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/OIDC-ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/OIDC-ID:aud": "sts.amazonaws.com",
          "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/OIDC-ID:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
        }
      }
    }
  ]
}
EOF

Note: Replace YOUR_AWS_ACCOUNT_ID with your AWS account ID, YOUR_AWS_REGION with your AWS Region, and OIDC-ID with your OIDC provider ID.

To create an IAM role that's named AmazonEKS_EBS_CSI_DriverRole, run the following create-role command:

aws iam create-role 
 --role-name AmazonEKS_EBS_CSI_DriverRole 
 --assume-role-policy-document file://"trust-policy.json"

To attach the CSI driver's AWS managed IAM policy to the IAM role, run the following attach-role-policy command:

aws iam attach-role-policy
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy 
--role-name AmazonEKS_EBS_CSI_DriverRole

To deploy the Amazon EBS CSI driver, use Kustomize, Helm, or an Amazon EKS managed add-on. For deployment instructions, see Installation on the GitHub website.
Test your Amazon EBS CSI driver with a sample application that uses dynamic provisioning for the Pods. The Amazon EBS CSI driver provisions EBS volumes on demand.

Amazon EFS CSI driver

Create an IAM role for the CSI driver

Complete the following steps for a cluster with worker nodes:

To use the AmazonEFSCSIDriverPolicy, see If using IAM roles for service accounts. Then, proceed to step 7.
Or, run the following command to download the IAM policy document from GitHub:

curl -o iam-policy-example.json https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/docs/iam-policy-example.json

To create an IAM policy, run the following create-policy command:

aws iam create-policy 
    --policy-name AmazonEKS_EFS_CSI_Driver_Policy 
    --policy-document file://iam-policy-example.json

To determine your cluster's OIDC provider ID, run the following describe-cluster command:

aws eks describe-cluster
 --name your_cluster_name
 --query "cluster.identity.oidc.issuer"
 --output text

Note: Replace your_cluster_name with your cluster name.

Create the following IAM trust policy that grants the AssumeRoleWithWebIdentity action to your Kubernetes service account:

cat <<EOF > trust-policy.json{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID:oidc-provider/oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/OIDC-ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/OIDC-ID:sub": "system:serviceaccount:kube-system:efs-csi-controller-sa"
        }
      }
    }
  ]
}
EOF

Note: Replace YOUR_AWS_ACCOUNT_ID with your account ID, YOUR_AWS_REGION with your Region, and OIDC-ID with your cluster's OIDC provider ID.

To create an IAM role, run the following create-role command:

aws iam create-role
  --role-name AmazonEKS_EFS_CSI_DriverRole 
  --assume-role-policy-document file://"trust-policy.json"

To attach your new IAM policy to the role, run the following attach-role-policy command:

aws iam attach-role-policy 
  --policy-arn arn:aws:iam::AWS_ACCOUNT_ID:policy/AmazonEKS_EFS_CSI_Driver_Policy 
  --role-name AmazonEKS_EFS_CSI_DriverRole

Note: Replace YOUR_AWS_ACCOUNT_ID with your account ID.

If you use a managed add-on, then proceed to Deploy the Amazon EFS CSI driver. To install a self-managed driver, complete steps 7-10. Save the following contents to a YAML file that's named efs-service-account.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: aws-efs-csi-driver
  name: efs-csi-controller-sa
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::AWS_ACCOUNT_ID:role/AmazonEKS_EFS_CSI_DriverRole

Note: Replace YOUR_AWS_ACCOUNT_ID with your account ID.

To create the Kubernetes service account on your cluster, run the following command:
```
kubectl apply -f efs-service-account.yaml
```
Note: The efs-csi-controller-sa Kubernetes service account has the annotation of the IAM role that you created.

To download the manifest from the public Amazon Elastic Container Registry (Amazon ECR) registry, run the following command:

kubectl kustomize "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-VERSION" > public-ecr-driver.yaml

Note: Replace VERSION with your release version number. It's a best practice to use the latest released version. For a list of active branches, see Branches on the GitHub website. You can use Helm instead of Kustomize to install the driver. For steps, see Deploy the driver on the GitHub website.

Edit the downloaded file to remove the following lines:

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: aws-efs-csi-driver
  name: efs-csi-controller-sa
  namespace: kube-system

Note: You don't need to include the preceding lines because you already created a service account in step 7.

Deploy the Amazon EFS CSI driver

To apply the manifest, run the following command:

kubectl apply -f public-ecr-driver.yaml

If your cluster doesn't contain nodes and contains only AWS Fargate Pods, then run the following command to deploy the driver to all Regions:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/deploy/kubernetes/base/csidriver.yaml

Create an EFS file system

Complete the following steps:

To get the virtual private cloud (VPC) ID of your Amazon EKS cluster, run the following describe-cluster command:

aws eks describe-cluster
 --name your_cluster_name
 --query "cluster.resourcesVpcConfig.vpcId"
 --output text

Note: Replace your_cluster_name with your cluster name.

To get the CIDR range for your VPC cluster, run the following describe-vpcs command:

aws ec2 describe-vpcs
 --vpc-ids YOUR_VPC_ID
 --query "Vpcs[].CidrBlock"
 --output text

Note: Replace the YOUR_VPC_ID with your VPC ID.

To create a security group that allows inbound network file system (NFS) traffic to your Amazon EFS mount points, run the following create-security-group command:

aws ec2 create-security-group
 --description efs-test-sg
 --group-name efs-sg
 --vpc-id YOUR_VPC_ID

Note: Replace YOUR_VPC_ID with your VPC ID. In the output, note the GroupId value.

To allow resources in your VPC to communicate with your EFS file system, run the following authorize-security-group-ingress command to add an NFS inbound rule:

aws ec2 authorize-security-group-ingress --group-id sg-abc --protocol tcp --port 2049 --cidr YOUR_VPC_CIDR

Note: Replace YOUR_VPC_CIDR with your VPC CIDR and sg-abc with your security group ID.

To create an EFS file system for your Amazon EKS cluster, run the following create-file-system command:

aws efs create-file-system --creation-token eks-efs

Note: Note the FileSystemId.

To create a mount target for Amazon EFS, run the following create-mount-target command:
```
aws efs create-mount-target --file-system-id FileSystemId --subnet-id SubnetID --security-group sg-abc
```
Note: Replace FileSystemId with your EFS file system ID, sg-abc with your security group ID, and SubnetID with your worker node subnet ID. It's a best practice to run the preceding command for each Availability Zone where your worker nodes run. All Amazon Elastic Compute Cloud (Amazon EC2) instances in the Availability Zones can use the file system. To create mount targets in multiple subnets, run the command for each subnet ID.