By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Pod cannot mount persistent volume from EFS

0

I have completed the steps recommended here: https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html. But my pod cannot start: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri" b'mount.nfs4: access denied by server while mounting 127.0.0.1:/

More details:

$kubectl describe pods/newrepoinfoextractor-5ff94cbdd7-hvcmg

Name:             newrepoinfoextractor-5ff94cbdd7-hvcmg
Namespace:        default
Priority:         0
Service Account:  default
Node:             ip-10-0-0-184.eu-central-1.compute.internal/10.0.0.184
Start Time:       Tue, 23 May 2023 12:27:08 +0200
Labels:           app=newrepoinfoextractor
                  pod-template-hash=5ff94cbdd7
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    ReplicaSet/newrepoinfoextractor-5ff94cbdd7
Containers:
  newrepoinfoextractor:
    Container ID:
    Image:         467521679414.dkr.ecr.eu-central-1.amazonaws.com/newrepoinfoextractor:master
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /newrepoinfoextractor
      --registry_address=consul:8500
      --registry=consul
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      ephemeral-storage:  20Gi
      memory:             3024Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  10Gi
      memory:             3024Mi
    Environment:
      RABBITMQ_URL:       amqp://admin:Ox3Wym0sQ6@rabbitmq.rabbitmq:5672
      TRACER_URL:         jaeger
      GRPC_GATEWAY_HOST:  https://grpcgateway2.codersrank.io
      WORK_DIR:           /repoinfoextractor
      DEBUG_MODE:         false
    Mounts:
      /mnt/repoinfoextractor from repoinfoextractor (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ffmcm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  repoinfoextractor:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  repoinfoextractor-pvc
    ReadOnly:   false
  kube-api-access-ffmcm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    31s                default-scheduler  Successfully assigned default/newrepoinfoextractor-5ff94cbdd7-hvcmg to ip-10-0-0-184.eu-central-1.compute.internal
  Warning  FailedMount  10s (x6 over 30s)  kubelet            MountVolume.SetUp failed for volume "repoinfoextractor-pv" : rpc error: code = Internal desc = Could not mount "fs-01xxxxxxxxxxxxxb6:/" at "/var/lib/kubelet/pods/f3630578-301c-45ce-b28b-34bb9f4e01d5/volumes/kubernetes.io~csi/repoinfoextractor-pv/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-0axxxxxxxxxxxxxec,tls,iam fs-01xxxxxxxxxxxxxb6:/ /var/lib/kubelet/pods/f3630578-301c-45ce-b28b-34bb9f4e01d5/volumes/kubernetes.io~csi/repoinfoextractor-pv/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].

kubectl get pv                                                                                                
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
pvc-b66b4822-df93-43cb-9cec-510bdb62d2d0   8Gi        RWO            Delete           Bound    rabbitmq/data-rabbitmq-2        gp2                     13d
pvc-d1663f89-ffa2-448d-b1c9-9e37bf09226f   8Gi        RWO            Delete           Bound    rabbitmq/data-rabbitmq-1        gp2                     13d
pvc-fb4f7db6-957c-401c-8edd-ead1696bee15   8Gi        RWO            Delete           Bound    rabbitmq/data-rabbitmq-0        gp2                     13d
repoinfoextractor-pv                       5Gi        RWX            Retain           Bound    default/repoinfoextractor-pvc   efs-sc                  5d3h

kubectl get pvc                                                                                      
NAME                    STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
repoinfoextractor-pvc   Bound    repoinfoextractor-pv   5Gi        RWX            efs-sc         5d3h

Yaml file for the deployment, VC, PVC, StorageClass, ServiceAccount:

---
apiVersion: v1
kind: Service
metadata:
  name: newrepoinfoextractor
  labels:
    app: newrepoinfoextractor
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: newrepoinfoextractor
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: newrepoinfoextractor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: newrepoinfoextractor
  template:
    metadata:
      labels:
        app: newrepoinfoextractor
    spec:
      containers:
        - name: newrepoinfoextractor
          command: [
            "/newrepoinfoextractor"
          ]
          image: xxxx.amazonaws.com/newrepoinfoextractor:master
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
              name: nre-extr-port
          volumeMounts:
            - name: repoinfoextractor
              mountPath: /mnt/repoinfoextractor
          resources:
            requests:
              memory: "3024Mi"
              cpu: "100m"
            limits:
              memory: "3024Mi"
      volumes:
        - name: repoinfoextractor
          persistentVolumeClaim:
            claimName: repoinfoextractor-pvc
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-016xxxxxxxxxxxxb6
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: aws-efs-csi-driver
  name: efs-csi-controller-sa
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::46xxxxxxxx14:role/AmazonEKS_EFS_CSI_DriverRole
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: repoinfoextractor-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  mountOptions:
    - tls
    - iam
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-01xxxxxxxxxxxxxb6::fsap-0axxxxxxxxxxxxxec
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: repoinfoextractor-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

In the policy simulator, everything looks ok: Enter image description here

I don't know how to move forward or how to debug what is wrong.

Topics
StorageContainers
Tags
Amazon Elastic File SystemAmazon Elastic Kubernetes Service
Language
English
rePost-User-8190828
asked a year ago3726 views
1 Answer
0

This issue can occur if your NFS client does not have permission to mount the file system.

Action to take

If you are attempting to mount the file system using IAM, make sure you are using the -o iam option in your mount command. This tells the EFS mount helper to pass your credentials to the EFS mount target. If you still don't have access, check your file system policy and your identity policy to ensure there are no DENY clauses that apply to your connection, and that there is at least one ALLOW clause that applies to the connection. For more information, see Using IAM to control file system data access and Creating file system policies.

Please have a look at the following link for more information: https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-mounting.html

Please inform me if my answer was satisfactory by accepting my response with an up-vote.

AWS
ART
answered a year ago
  • rePost-User-8190828
    a year ago

    According to the logs -o iam is used:

    Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-0axxxxxxxxxxxxxec,tls,iam fs-01xxxxxxxxxxxxxb6:/ /var/lib/kubelet/pods/f3630578-301c-45ce-b28b-34bb9f4e01d5/volumes/kubernetes.io~csi/repoinfoextractor-pv/mount

    I check all of the policies and non of them has deny effect.

  • ART
    a year ago

    Request that you please re-organize the command passed to reflect <-o iam> and then follow up with other arguments. I see that your command does have the <-o> but it has its argument reflect the accesspoint <-o accesspoint=fsap-0axxxxxxxxxxxxxec> and not the <iam>. Can you please make the change and see if it work? Also, can you please check for an ALLOW policy as per the article?

  • ART
    a year ago

    Request that you please re-organize the command passed to reflect <-o iam> and then follow up with other arguments. I see that your command does have the <-o> but it has its argument reflect the accesspoint <-o accesspoint=fsap-0axxxxxxxxxxxxxec> and not the <iam>. Can you please make the change and see if it work? Also, can you please check for an ALLOW policy as per the article?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content