Kubernetes Persistent Volume never created by local path provisioner

Question

In my Kubernetes cluster, Rancher never creates Persistent Volumes after creating a Persistent Volume Claim and applying a Pod.

Solution/Work around available under second update.

The cluster has been installed with Kubespray. The configuration for local path provisioning in inventory/myclster/group_vars/k8s-cluster/addons.yml:

# Rancher Local Path Provisioner
local_path_provisioner_enabled: true
# local_path_provisioner_namespace: "local-path-storage"
# local_path_provisioner_storage_class: "local-path"
# local_path_provisioner_reclaim_policy: Delete
# local_path_provisioner_claim_root: /opt/local-path-provisioner/
# local_path_provisioner_debug: false
# local_path_provisioner_image_repo: "rancher/local-path-provisioner"
# local_path_provisioner_image_tag: "v0.0.14"
# local_path_provisioner_helper_image_repo: "busybox"
# local_path_provisioner_helper_image_tag: "latest"

# Local volume provisioner deployment
local_volume_provisioner_enabled: false
# local_volume_provisioner_namespace: kube-system
# local_volume_provisioner_nodelabels:
#   - kubernetes.io/hostname
#   - topology.kubernetes.io/region
#   - topology.kubernetes.io/zone
# local_volume_provisioner_storage_classes:
#   local-storage:
#     host_dir: /mnt/disks
#     mount_dir: /mnt/disks
#     volume_mode: Filesystem
#     fs_type: ext4
#   fast-disks:
#     host_dir: /mnt/fast-disks
#     mount_dir: /mnt/fast-disks
#     block_cleaner_command:
#       - "/scripts/shred.sh"
#       - "2"
#     volume_mode: Filesystem
#     fs_type: ext4

Steps to recreate the problem:

Create PVC:
kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pvc.yaml

Result: Created PVC, name local-path-pvc, status Pending, storage class local-path

Create Pod
kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod.yaml

Result:
Created Pod, name create-pvc-123, status Waiting:ContainerCreating.

Describing Pod with kubectl:

Events:  
  Type     Reason            Age        From  Message
  ----     ------            ----       ----  -------
  Warning  FailedScheduling  <unknown>        error while running "VolumeBinding" prebind plugin for pod "create-pvc-123": Failed to bind volumes: timed out waiting for the condition

I have tried different charts, and in all cases no Persistent Volume has been created. The ServiceAccount local-path-provisioner-service-account exists. The Deployment of the local path provisioner has one Pod.

UPDATE
On the server the logs contains several errors, sudo journalctl -xeu kubelet | grep 'fail':

...
Oct 12 16:53:36 node1 kubelet[274306]: E1012 16:53:36.000246  274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/71b44438-fadb-4859-a788-8d911dfab2db-script podName:71b44438-fadb-4859-a788-8d911dfab2db nodeName:}" failed. No retries permitted until 2020-10-12 16:54:40.000164134 +0200 CEST m=+9380.643933974 (durationBeforeRetry 1m4s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/71b44438-fadb-4859-a788-8d911dfab2db-script\") pod \"create-pvc-80d115d9-98fd-4fcd-9e41-55b74f809efb\" (UID: \"71b44438-fadb-4859-a788-8d911dfab2db\") : configmap references non-existent config key: setup"
Oct 12 16:53:36 node1 kubelet[274306]: E1012 16:53:36.404015  274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script podName:424b196e-5132-479a-8b95-63e41e0ea124 nodeName:}" failed. No retries permitted until 2020-10-12 16:54:40.403980548 +0200 CEST m=+9381.047750378 (durationBeforeRetry 1m4s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script\") pod \"create-pvc-3b132d90-8812-4391-bc29-966ee47bee0d\" (UID: \"424b196e-5132-479a-8b95-63e41e0ea124\") : configmap references non-existent config key: setup"
Oct 12 16:54:40 node1 kubelet[274306]: E1012 16:54:40.464999  274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script podName:424b196e-5132-479a-8b95-63e41e0ea124 nodeName:}" failed. No retries permitted until 2020-10-12 16:56:42.464936126 +0200 CEST m=+9503.108706016 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script\") pod \"create-pvc-3b132d90-8812-4391-bc29-966ee47bee0d\" (UID: \"424b196e-5132-479a-8b95-63e41e0ea124\") : configmap references non-existent config key: setup"

UPDATE - solution?
I changed the ConfigMap 'local-path-config' as described in the docs.
However, the jinja template in Kubespray lacks the properties 'setup' and 'teardown' in the configuration.

When I added de setup and teardown properties Kubernetes created the PV and the Pod started.

What's the reason Kubespray doesn't provide these properties in the template?

Answer 1

Default version that kubespray should use is local_path_provisioner_image_tag: "v0.0.14". I suppose the source for the template is https://github.com/rancher/local-path-provisioner/blob/v0.0.14/deploy/example-config.yaml. It doesn't have setup and teardown properties, they were introduced in v0.0.15.