`mkdir` returns successfully but doesn't work in kubernetes (minikube) shared volume

Question

I am trying to set up a shared volume in a minikube Kubernetes cluster to allow multiple pods to communicate with each other. What is configured is:

• A PVC using the nfs-server-provisioner dynamic provisioner
• Multiple Pods (some are jobs) that mount the PVC

The goal is to have an init container in each pod that creates a directory on startup using the Pod's name as the directory name, and have a job scan that directory and do some stuff.

I have this configured, and no errors are thrown, but the directory isn't created.

When trying to do this manually I see some strange behavior; mkdir returns a non-error code but doesn't do anything:

< ssh into pod >

user@802542b3ccb195b001258094dc543606-1601299620-zcszs:~$ ls -al /output/
total 8
drwxrwxrwx 2 user   users 4096 Sep 28 13:28 .
drwxr-xr-x 1 root   root  4096 Sep 28 13:27 ..
user@802542b3ccb195b001258094dc543606-1601299620-zcszs:~$ mkdir /output/test
user@802542b3ccb195b001258094dc543606-1601299620-zcszs:~$ echo $#
0
user@802542b3ccb195b001258094dc543606-1601299620-zcszs:~$ ls -al /output/
total 8
drwxrwxrwx 2 user   users 4096 Sep 28 13:28 .
drwxr-xr-x 1 root   root  4096 Sep 28 13:27 ..
user@802542b3ccb195b001258094dc543606-1601299620-zcszs:~$ 

I am able to touch files:

user@802542b3ccb195b001258094dc543606-1601299740-bw2hj:~$ ls -al /output/
total 8
drwxrwxrwx 2 user   users 4096 Sep 28 13:29 .
drwxr-xr-x 1 root   root  4096 Sep 28 13:29 ..          
user@802542b3ccb195b001258094dc543606-1601299740-bw2hj:~$ touch /output/test
user@802542b3ccb195b001258094dc543606-1601299740-bw2hj:~$ ls -al /output/
total 8
drwxrwxrwx 2 user   users 4096 Sep 28 13:29 .
drwxr-xr-x 1 root   root  4096 Sep 28 13:29 ..
-rw-r--r-- 1 user   users    0 Sep 28 13:29 test
user@802542b3ccb195b001258094dc543606-1601299740-bw2hj:~$ 

Here is the nfs mount:

Filesystem                                                      Size  Used Avail Use% Mounted on
10.110.46.205:/export/pvc-2e433dc6-018d-11eb-be1a-0242766f1f7c  252G  134G  107G  56% /output

The same behavior is observed when using regular volumes. I am using the Docker driver but also observed this with the virtualbox driver. Is this a minikube issue? I would expect mkdir to error out if it can't complete.

minikube version: v1.13.1
commit: 1fd1f67f338cbab4b3e5a6e4c71c551f522ca138-dirty