How do I install the Amazon S3 CSI driver on my Amazon EKS clusters?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Prerequisites:

• Install the kubectl and eksctl command line tools.
• Create an AWS Identity and Access Management( IAM) OpenID Connect (OIDC) provider for your cluster.
• Use IAM roles for service accounts (IRSA) because you can't use EKS Pod Identity with the Amazon S3 CSI driver.

To get your OIDC issuer ID, run the following describe-cluster AWS CLI command:

aws eks describe-cluster --name your_cluster_name --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5

Note: Replace your_cluster_name with your Amazon EKS cluster name.

To check whether you configured an IAM OIDC provider, run the following list-open-id-connect-providers command:

aws iam list-open-id-connect-providers | grep your_OIDC_ID | cut -d "/" -f4

Note: Replace YOUR_OIDC_ID with your OIDC ID. If the output is empty, then you must create an IAM OIDC provider.

To create the IAM OIDC provider, run the following command:

eksctl utils associate-iam-oidc-provider --cluster your_cluster_name --approve

Note: Replace your_cluster_name with your cluster name.

Deploy the Amazon S3 CSI driver

Complete the following steps:

1. Create an IAM policy based on your requirements.
   General purpose bucket policy:

   cat <<EOF > iam-policy.json
   {
      "Version": "2012-10-17",
      "Statement": [
           {
               "Sid": "MountpointFullBucketAccess",
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::your_bucket_name"
               ]
           },
           {
               "Sid": "MountpointFullObjectAccess",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::your_bucket_name/*"
               ]
           }
      ]
   }
   EOF

   Note: Replace your_bucket_name with your bucket name.
   Directory bucket policy:

   cat <<EOF > iam-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": "s3express:CreateSession",
               "Resource": "arn:aws:s3express:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:bucket/your_bucket_name--az_id--x-s3"
           }
       ]
   }
   EOF

   Note: Replace YOUR_AWS_REGION with your AWS Region, YOUR_AWS_ACCOUNT_ID with your AWS account, your_bucket_name with your bucket name, and az_id with your Availability Zone.
2. Run the following create-policy command to create an IAM policy that's named AmazonS3CSIDriverPolicy:

   aws iam create-policy --policy-name AmazonS3CSIDriverPolicy --policy-document file://iam-policy.json

   Note: Replace iam-policy.json with your IAM policy JSON file.
3. Create the following IAM trust policy:

   cat <<EOF > trust-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Principal": {
             "Federated": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID:oidc-provider/oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/your_OIDC_ID"
           },
           "Action": "sts:AssumeRoleWithWebIdentity",
           "Condition": {
             "StringLike": {
               "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/your_OIDC_ID:sub": "system:serviceaccount:kube-system:s3-csi-*",
               "oidc.eks.YOUR_AWS_REGION.amazonaws.com/id/your_OIDC_ID:aud": "sts.amazonaws.com"
             }
           }
         }
       ]
   }
   EOF

   Note: Replace YOUR_AWS_ACCOUNT_ID with your account, YOUR_AWS_REGION with your Region and YOUR_OIDC_ID with your OIDC ID.
4. Run the following create-role command to create an IAM role that's named AmazonEKS_S3_CSI_DriverRole:

   aws iam create-role --role-name AmazonEKS_S3_CSI_DriverRole  --assume-role-policy-document file://"trust-policy.json"

   Note: Replace trust-policy.json with your IAM trust policy JSON file.
5. Run the following attach-role-policy command to attach the policy to the IAM role:

   aws iam attach-role-policy  --policy-arn arn:aws:iam::YOUR_AWS_ACCOUNT_ID:policy/AmazonS3CSIDriverPolicy --role-name AmazonEKS_S3_CSI_DriverRole

   Note: Replace YOUR_AWS_ACCOUNT_ID with your AWS account ID.
6. Run the following create-addon command to deploy the Amazon S3 CSI driver on your cluster:

   aws eks  create-addon --cluster-name your_cluster_name --addon-name aws-mountpoint-s3-csi-driver --service-account-role-arn arn:aws:iam::YOUR_AWS_ACCOUNT_ID:role/AmazonEKS_S3_CSI_DriverRole

   Note: Replace your_cluster_name with your cluster name, and YOUR_AWS_ACCOUNT_ID with your account ID. To install the Amazon S3 CSI driver with Kustomize or Helm, see Installation on the GitHub website.

To allow users to overwrite existing files, add the --allow-overwrite flag to the persistent volume. You must add all writes sequentially from the beginning of the file. You can't append files. Instead, you must replace existing content with new content. However, for directory buckets in the Amazon S3 Express One Zone storage class, you can append existing files. To do this, add the --incremental-upload flag to the persistent volume. Also, start from the end of the file to sequentially add all writes. For more information, see Reading and writing files and Configure on the GitHub website.

The Amazon S3 CSI driver mount point uses only static provisioning. You can't use dynamic provisioning, or create new buckets. Volumes that you mount on the mount point can't use all POSIX file system features. For more information, see Mountpoint for Amazon S3 file system behavior on the GitHub website. If your applications require complete file systems, then it's a best practice to use Amazon Elastic File System (Amazon EFS) or Amazon FSx.

Test the Amazon S3 CSI driver

To test the Amazon S3 CSI driver, deploy a sample application that uses static provisioning for the Pods. For steps and examples, see Static provisioning example on the GitHub website.

Troubleshoot issues with the Amazon S3 CSI driver

Your Amazon S3 CSI driver isn't working as expected

To verify that the Amazon S3 CSI driver node Pods are running, run the following command:

kubectl get all -l app.kubernetes.io/name=aws-mountpoint-s3-csi-driver -n kube-system

To review the logs from the Amazon S3 CSI driver Pods, run the following command:

kubectl logs daemonset/s3-csi-node -n kube-system -c s3-plugin:

Make sure that the Amazon S3 bucket is in the same Region as the Amazon EKS cluster. If the bucket and cluster are in different Regions, then update the mountOptions to include the Region.

Example configuration:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: s3-pv-1
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  csi:
    driver: s3.csi.aws.com
    volumeAttributes:
      bucketName: bucket name
    volumeHandle: s3-csi-driver-volume
  mountOptions:
  - region us-west-2 
  - allow-other     
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

If the Amazon S3 CSI driver doesn't work on Bottlerocket Amazon Machine Images (AMIs), then run the following command to check the container image version:

kubectl get ds s3-csi-node -n kube-system -ojsonpath="{range .spec.template.spec.containers[*]}{.image}{'\n'}{end}"

In the command's output, make sure that the container image version is greater than or equal to v1.4.0. It's a best practice to use the latest Amazon S3 CSI driver version. For the latest version, see mountpoint-s3-csi-driver Releases on the GitHub website.

Users in your container can't access the Amazon S3 bucket

By default, only the user that mounted the S3 bucket can access your mounted directory. This occurs even when you configure permissions and ownership settings to allow other users in the container to access the bucket.

To allow other non-root users to access your mounted directory, update mountOptions to include the --allow-other option. To allow the root user to access your mounted directory as a different user, use --allow-root.

Example policy:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: s3-pv-1
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  csi:
    driver: s3.csi.aws.com
    volumeAttributes:
      bucketName: bucket name
    volumeHandle: s3-csi-driver-volume
  mountOptions:
  - region us-west-2
  - allow-other                        
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

You receive the "Volume capability not supported" error

When you use the Amazon S3 CSI driver, you might receive an error message that's similar to the following example:

"Warning FailedMount 39s (x8 over 104s) kubelet MountVolume.SetUp failed for volume "s3-pv-1" : rpc error: code = InvalidArgument desc = Volume capability not supported"

You receive the preceding error when you use an access mode that Amazon S3 CSI driver doesn't support, such as ReadWriteOncePod or ReadWriteOnce. You can use only ReadWriteMany or ReadOnlyMany access modes.