Kubernetes Vsphere Provisioner: Failed to provision volume with StorageClass "fast": Cloud provider not initialized properly

Question

Summary I have provisioned kubernetes on several vcenter vms using kubespray and everything there is working perfectly. However I am running into trouble with setting up Vsphere as the cloud provider and dynamically provisioning persistent volumes. I have tried following this guide on setting up Vsphere as a provider. All of the pods seem to launch:

vsphere-cloud-controller-manager-5h8vr        1/1     Running   0          77m
vsphere-csi-controller-0                      5/5     Running   0          52m
vsphere-csi-node-9gcn5                        3/3     Running   0          23m
vsphere-csi-node-lcmvn                        3/3     Running   0          23m
vsphere-csi-node-pgbcj                        3/3     Running   0          23m
vsphere-csi-node-rvx4m                        3/3     Running   0          23m

However if I describe the pvc I get this error:

  Warning  ProvisioningFailed  64s (x30 over 52m)  persistentvolume-controller  Failed to provision volume with StorageClass "fast": Cloud provider not initialized properly

The only error I can find is from the vsphere-csi-nodes:

k logs vsphere-csi-node-pgbcj -c vsphere-csi-node

I0709 18:03:55.515251       1 service.go:88] configured: csi.vsphere.vmware.com with map[mode:node]
time="2020-07-09T18:03:55Z" level=info msg="identity service registered"
time="2020-07-09T18:03:55Z" level=info msg="node service registered"
time="2020-07-09T18:03:55Z" level=info msg=serving endpoint="unix:///csi/csi.sock"
I0709 18:03:57.645866       1 config.go:261] GetCnsconfig called with cfgPath: /etc/cloud/csi-vsphere.conf
I0709 18:03:57.645925       1 config.go:265] Could not stat /etc/cloud/csi-vsphere.conf, reading config params from env
E0709 18:03:57.645968       1 config.go:202] No Virtual Center hosts defined
E0709 18:03:57.645983       1 config.go:269] Failed to get config params from env. Err: No Virtual Center hosts defined
I0709 18:03:57.645995       1 node.go:471] Config file not provided to node daemonset. Assuming non-topology aware cluster.

Looking through "secrets" it looks as though all of that information was propagated correctly.

Pre-reqs in place

1. VM hardware-version >15
2. UUID enabled on all vms
3. Vsphere >6.7

Environment

1. Single Master
2. Three Nodes
3. Host OS: Debian 10.04 64bit
4. Kubespray: latest
5. Kubernetes: 1.18.5
6. VM "hardware version": 17
7. Esxi 7
8. Vcenter 7
9. CNI Weave

Config Files

group_vars/all.yml

cloud_provider: "external"

external_cloud_provider: "vsphere"

vsphere.conf

# Vsphere provider
external_vsphere_vcenter_ip: "serveriphere"
external_vsphere_vcenter_port: "443"
external_vsphere_insecure: "true"
external_vsphere_user: "[email protected]"
external_vsphere_password: "indestructiblepasswordhere"
external_vsphere_datacenter: "LAB"
external_vsphere_kubernetes_cluster_id: "k8"

vsphere_cloud_controller_image_tag: "latest"
vsphere_syncer_image_tag: "v1.0.2"
vsphere_csi_attacher_image_tag: "v1.1.1"
vsphere_csi_controller: "v1.0.2"
vsphere_csi_liveness_prove_image_tag: "v1.1.0"
vsphere_csi_provisioner_image_tag: "v1.2.2"
vsphere_csi_node_driver_registrar_image_tag: "v1.1.0"
vsphere_csi_driver_image_tag: "v1.0.2"

vsphere_csi_controller_replicas: 1


vsphere_csi_enabled: true

Storage Class

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/vsphere-volume
parameters:
  datastore: LAB
  diskformat: thin
  fsttype: ext4

Persistent Volume Claim

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: fast