I am deploying RabbitMQ on AWS EKS Cluster using helm chart. I am using EFS as storage class for the stateful pod. After deployment the stateful pod shows EFS volume assigned and I see PV, PVC created and PV gets into BOUND state. But after that, the pod gets stuck into "Init" state. and after around 2 minutes throws error --
"Warning FailedMount 24s (x5 over 8m32s) kubelet MountVolume.SetUp failed for volume "pvc-7478e160-bf20-48f1-811e-1c1b57c5c2f6" : rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I checked the EC2 instance on which the the volume was PV assigned and in /var/log/messages, it showed this error --
"Feb 21 01:07:34 kubelet: E0221 01:07:34.056582 3227 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/efs.csi.aws.com^fs-03a35a89a0176bd23::fsap-04862d48fb121b44a podName: nodeName:}" failed. No retries permitted until 2024-02-21 01:09:36.056447805 +0000 UTC m=+91661.321657916 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-7478e160-bf20-48f1-811e-1c1b57c5c2f6" (UniqueName: "kubernetes.io/csi/efs.csi.aws.com^fs-03a35a89a0176bd23::fsap-04862d48fb121b44a") pod "rabbitmq-server-0" (UID: "9eef38f3-fabc-4b60-9efa-f5e97fea7557") : rpc error: code = DeadlineExceeded desc = context deadline exceeded"
RabbitMQ POD Events --
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 36m default-scheduler 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
Normal Scheduled 36m default-scheduler Successfully assigned default/tmgm-risk-uat-us-aws-server-0 to ip-<IP HIDDEN>.ec2.internal
Warning FailedMount 17m kubelet MountVolume.SetUp failed for volume "pvc-7478e160-bf20-48f1-811e-1c1b57c5c2f6" : rpc error: code = Internal desc = Could not mount "fs-03a35a89a0176bd23:/" at "/var/lib/kubelet/pods/9eef38f3-fabc-4b60-9efa-f5e97fea7557/volumes/kubernetes.io~csi/pvc-7478e160-bf20-48f1-811e-1c1b57c5c2f6/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-04862d48fb121b44a,tls fs-03a35a89a0176bd23:/ /var/lib/kubelet/pods/9eef38f3-fabc-4b60-9efa-f5e97fea7557/volumes/kubernetes.io~csi/pvc-7478e160-bf20-48f1-811e-1c1b57c5c2f6/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
Mount attempt 1/3 failed due to timeout after 15 sec, wait 0 sec before next attempt.
Mount attempt 2/3 failed due to timeout after 15 sec, wait 0 sec before next attempt.
b'mount.nfs4: mount point /var/lib/kubelet/pods/9eef38f3-fabc-4b60-9efa-f5e97fea7557/volumes/kubernetes.io~csi/pvc-7478e160-bf20-48f1-811e-1c1b57c5c2f6/mount does not exist'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
Warning FailedMount 85s (x12 over 34m) kubelet MountVolume.SetUp failed for volume "pvc-7478e160-bf20-48f1-811e-1c1b57c5c2f6" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
efs-csi-node logs --
E0221 03:04:44.030444 1 mount_linux.go:231] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-051e46ad6dcaa52f9,tls,iam fs-03a35a89a0176bd23:/ /var/lib/kubelet/pods/5b0696a1-b2e3-4f95-bac6-3056b6a0eb2b/volumes/kubernetes.io~csi/pvc-06c6efea-b9ee-46ae-8adf-000dc8e0791d/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
Mount attempt 1/3 failed due to timeout after 15 sec, wait 0 sec before next attempt.
Mount attempt 2/3 failed due to timeout after 15 sec, wait 0 sec before next attempt.
b'mount.nfs4: Connection timed out'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
E0221 03:04:44.030539 1 driver.go:106] GRPC error: rpc error: code = Internal desc = Could not mount "fs-03a35a89a0176bd23:/" at "/var/lib/kubelet/pods/5b0696a1-b2e3-4f95-bac6-3056b6a0eb2b/volumes/kubernetes.io~csi/pvc-06c6efea-b9ee-46ae-8adf-000dc8e0791d/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-051e46ad6dcaa52f9,tls,iam fs-03a35a89a0176bd23:/ /var/lib/kubelet/pods/5b0696a1-b2e3-4f95-bac6-3056b6a0eb2b/volumes/kubernetes.io~csi/pvc-06c6efea-b9ee-46ae-8adf-000dc8e0791d/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
Mount attempt 1/3 failed due to timeout after 15 sec, wait 0 sec before next attempt.
Mount attempt 2/3 failed due to timeout after 15 sec, wait 0 sec before next attempt.
b'mount.nfs4: Connection timed out'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
W0221 03:05:14.884369 1 node.go:163] Use of 'tls' under mountOptions is deprecated with this driver since tls is enabled by default. To disable it, set encrypt in transit in the volumeContext, e.g. 'encryptInTransit: true'