Permission denied writing artifacts to an NFS-mounted PVC

Question

I'm attempting to write MLflow artifacts to an NFS-mounted PVC. It's a new PVC mounting at /opt/mlflow, but MLflow seems to have permission writing to it. The specific error I'm getting is

PermissionError: [Errno 13] Permission denied: '/opt/mlflow'

I ran the same deployment with an S3-backed artifact store, and that worked just fine. That was on my home computer though, and I don't have the ability to do that at work. The MLflow documentation seems to indicate that I don't need any special syntax for NFS mounts.

Answer 1

Independent of MLflow you can approach this in a standard file permission way.

1. Exec into your pod and view the permissions at that file path

kubectl exec -it <pod>

ls -l /opt/mlflow

2. Within your pod/container see what user you are running as

whoami

If your user doesn't have access to that filepath, then you could adjust the file permissions by mounting the pvc into a different pod that runs under a user with permission to adjust them. Or you could try using fsGroup to control the permissions of the files mounted, which you can read more about here.