Pod cannot mount persistent volume from EFS
I have completed the steps recommended here: https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html. But my pod cannot start: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri" b'mount.nfs4: access denied by server while mounting 127.0.0.1:/
More details:
$kubectl describe pods/newrepoinfoextractor-5ff94cbdd7-hvcmg
Name: newrepoinfoextractor-5ff94cbdd7-hvcmg
Namespace: default
Priority: 0
Service Account: default
Node: ip-10-0-0-184.eu-central-1.compute.internal/10.0.0.184
Start Time: Tue, 23 May 2023 12:27:08 +0200
Labels: app=newrepoinfoextractor
pod-template-hash=5ff94cbdd7
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/newrepoinfoextractor-5ff94cbdd7
Containers:
newrepoinfoextractor:
Container ID:
Image: 467521679414.dkr.ecr.eu-central-1.amazonaws.com/newrepoinfoextractor:master
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/newrepoinfoextractor
--registry_address=consul:8500
--registry=consul
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
ephemeral-storage: 20Gi
memory: 3024Mi
Requests:
cpu: 100m
ephemeral-storage: 10Gi
memory: 3024Mi
Environment:
RABBITMQ_URL: amqp://admin:Ox3Wym0sQ6@rabbitmq.rabbitmq:5672
TRACER_URL: jaeger
GRPC_GATEWAY_HOST: https://grpcgateway2.codersrank.io
WORK_DIR: /repoinfoextractor
DEBUG_MODE: false
Mounts:
/mnt/repoinfoextractor from repoinfoextractor (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ffmcm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
repoinfoextractor:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: repoinfoextractor-pvc
ReadOnly: false
kube-api-access-ffmcm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 31s default-scheduler Successfully assigned default/newrepoinfoextractor-5ff94cbdd7-hvcmg to ip-10-0-0-184.eu-central-1.compute.internal
Warning FailedMount 10s (x6 over 30s) kubelet MountVolume.SetUp failed for volume "repoinfoextractor-pv" : rpc error: code = Internal desc = Could not mount "fs-01xxxxxxxxxxxxxb6:/" at "/var/lib/kubelet/pods/f3630578-301c-45ce-b28b-34bb9f4e01d5/volumes/kubernetes.io~csi/repoinfoextractor-pv/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-0axxxxxxxxxxxxxec,tls,iam fs-01xxxxxxxxxxxxxb6:/ /var/lib/kubelet/pods/f3630578-301c-45ce-b28b-34bb9f4e01d5/volumes/kubernetes.io~csi/repoinfoextractor-pv/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-b66b4822-df93-43cb-9cec-510bdb62d2d0 8Gi RWO Delete Bound rabbitmq/data-rabbitmq-2 gp2 13d
pvc-d1663f89-ffa2-448d-b1c9-9e37bf09226f 8Gi RWO Delete Bound rabbitmq/data-rabbitmq-1 gp2 13d
pvc-fb4f7db6-957c-401c-8edd-ead1696bee15 8Gi RWO Delete Bound rabbitmq/data-rabbitmq-0 gp2 13d
repoinfoextractor-pv 5Gi RWX Retain Bound default/repoinfoextractor-pvc efs-sc 5d3h
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
repoinfoextractor-pvc Bound repoinfoextractor-pv 5Gi RWX efs-sc 5d3h
Yaml file for the deployment, VC, PVC, StorageClass, ServiceAccount:
---
apiVersion: v1
kind: Service
metadata:
name: newrepoinfoextractor
labels:
app: newrepoinfoextractor
spec:
ports:
- port: 8080
protocol: TCP
selector:
app: newrepoinfoextractor
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: default
name: newrepoinfoextractor
spec:
replicas: 1
selector:
matchLabels:
app: newrepoinfoextractor
template:
metadata:
labels:
app: newrepoinfoextractor
spec:
containers:
- name: newrepoinfoextractor
command: [
"/newrepoinfoextractor"
]
image: xxxx.amazonaws.com/newrepoinfoextractor:master
imagePullPolicy: Always
ports:
- containerPort: 8080
name: nre-extr-port
volumeMounts:
- name: repoinfoextractor
mountPath: /mnt/repoinfoextractor
resources:
requests:
memory: "3024Mi"
cpu: "100m"
limits:
memory: "3024Mi"
volumes:
- name: repoinfoextractor
persistentVolumeClaim:
claimName: repoinfoextractor-pvc
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
parameters:
provisioningMode: efs-ap
fileSystemId: fs-016xxxxxxxxxxxxb6
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/name: aws-efs-csi-driver
name: efs-csi-controller-sa
namespace: kube-system
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::46xxxxxxxx14:role/AmazonEKS_EFS_CSI_DriverRole
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: repoinfoextractor-pv
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
mountOptions:
- tls
- iam
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
csi:
driver: efs.csi.aws.com
volumeHandle: fs-01xxxxxxxxxxxxxb6::fsap-0axxxxxxxxxxxxxec
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: repoinfoextractor-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-sc
resources:
requests:
storage: 5Gi
In the policy simulator, everything looks ok:
I don't know how to move forward or how to debug what is wrong.
- Mais recentes
- Mais votos
- Mais comentários
This issue can occur if your NFS client does not have permission to mount the file system.
Action to take
If you are attempting to mount the file system using IAM, make sure you are using the -o iam option in your mount command. This tells the EFS mount helper to pass your credentials to the EFS mount target. If you still don't have access, check your file system policy and your identity policy to ensure there are no DENY clauses that apply to your connection, and that there is at least one ALLOW clause that applies to the connection. For more information, see Using IAM to control file system data access and Creating file system policies.
Please have a look at the following link for more information: https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-mounting.html
Please inform me if my answer was satisfactory by accepting my response with an up-vote.
Conteúdo relevante
- AWS OFICIALAtualizada há 3 anos
- AWS OFICIALAtualizada há 2 anos
- AWS OFICIALAtualizada há um ano
According to the logs
-o iam
is used:I check all of the policies and non of them has
deny
effect.Request that you please re-organize the command passed to reflect <-o iam> and then follow up with other arguments. I see that your command does have the <-o> but it has its argument reflect the accesspoint <-o accesspoint=fsap-0axxxxxxxxxxxxxec> and not the <iam>. Can you please make the change and see if it work? Also, can you please check for an ALLOW policy as per the article?
Request that you please re-organize the command passed to reflect <-o iam> and then follow up with other arguments. I see that your command does have the <-o> but it has its argument reflect the accesspoint <-o accesspoint=fsap-0axxxxxxxxxxxxxec> and not the <iam>. Can you please make the change and see if it work? Also, can you please check for an ALLOW policy as per the article?