Unable to mount PVC created by OpenEBS on pods on Kubernetes bare-metal deployment

Question

I am facing issues while mounting pvc on pods with openebs installed on bare-metal kubernetes cluster created with RKE.

Expected Behavior

PVC's should be mounted on pods without issues.

Current Behavior

Pods unable to mount PVC's:

  Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Warning  FailedScheduling        2m9s (x23 over 2m45s)  default-scheduler        pod has unbound PersistentVolumeClaims (repeated 4 times)
  Normal   Scheduled               2m8s                   default-scheduler        Successfully assigned default/minio-deployment-64d7c79464-966jr to 192.168.1.21
  Normal   SuccessfulAttachVolume  2m8s                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff"
  Warning  FailedMount             84s (x4 over 102s)     kubelet, 192.168.1.21    MountVolume.WaitForAttach failed for volume "pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 10.43.227.122:3260 output: iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: No portals found
, err exit status 21
  Warning  FailedMount  24s (x4 over 80s)  kubelet, 192.168.1.21  MountVolume.WaitForAttach failed for volume "pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target: iqn.2016-09.com.openebs.jiva:pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff, portal: 10.43.227.122,3260].
iscsiadm: initiator reported error (12 - iSCSI driver not found. Please make sure it is loaded, and retry the operation)
iscsiadm: Could not log into all portals
Logging in to [iface: default, target: iqn.2016-09.com.openebs.jiva:pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff, portal: 10.43.227.122,3260] (multiple)
 (exit status 12)
  Warning  FailedMount  2s  kubelet, 192.168.1.21  Unable to mount volumes for pod "minio-deployment-64d7c79464-966jr_default(640263d0-ec99-11e8-85c9-b06ebfd124ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"minio-deployment-64d7c79464-966jr". list of unmounted volumes=[storage]. list of unattached volumes=[storage default-token-9n8pn]

Steps to Reproduce

1. Install openebs with helm.
2. Create a pvc with storage class as openebs-standalone
3. Create pod and try to mount the PVC.

kubectl get pvc:

root@an4:/home/rke-k8s# kubectl get pvc
NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
docker-private-registry-docker-registry   Bound    pvc-58cf63c1-ec95-11e8-9b5d-2cfda16d3cfd   10Gi       RWO            openebs-standalone   22m

Update

When I tried the sample minio deployment, here's what I have observed:

1. PVC creation took around 1-2 minutes.
2. Mounting PVC to the pod took around 1 hour.
3. Storage class used for this was openebs-standard.

Any reason for this? It is on-prem cluster deployment.

Answer 1

Well, this issue was documented in the troubleshooting guide - https://docs.openebs.io/docs/next/tsgiscsi.html

Answer 2

This is the issue with openebs and already been opened with team. Fix is still pending, you can track the issue here:

https://github.com/openebs/openebs/issues/1688

There is step by step instruction how to debug the issue. Hope this helps.