Openshift missing permissions to create a file

Question

The spring boot application is deployed on openshift 4. This application needs to create a file on the nfs-share. The openshift container has configured a volume mount on the type NFS. The container on openshift creates a pod with random userid as

sh-4.2$ id
uid=1031290500(1031290500) gid=0(root) groups=0(root),1031290500

The mount point is /nfs/abc

sh-4.2$ ls -la /nfs/
ls: cannot access /nfs/abc: Permission denied
total 0
drwxr-xr-x. 1 root root 29 Nov 25 09:34 .
drwxr-xr-x. 1 root root 50 Nov 25 10:09 ..
d?????????? ? ?    ?     ?            ? abc

on the docker image I created a user "technical" with uid= gid=48760 as shown below.

FROM quay.repository
MAINTAINER developer

LABEL description="abc image" \
      name="abc" \
      version="1.0"

ARG APP_HOME=/opt/app
ARG PORT=8080

ENV JAR=app.jar \
    SPRING_PROFILES_ACTIVE=default \
    JAVA_OPTS=""

    
RUN mkdir $APP_HOME 

ADD $JAR $APP_HOME/

WORKDIR $APP_HOME
EXPOSE $PORT
ENTRYPOINT java $JAVA_OPTS -Dspring.profiles.active=$SPRING_PROFILES_ACTIVE -jar $JAR

my deployment config file is as shown below

 spec:
      volumes:
        - name: bad-import-file
          persistentVolumeClaim:
            claimName: nfs-test-pvc
      containers:
        - resources:
            limits:
              cpu: '1'
              memory: 1Gi
            requests:
              cpu: 500m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          name: abc
          env:
            - name: SPRING_PROFILES_ACTIVE
              valueFrom:
                configMapKeyRef:
                  name: abc-configmap
                  key: spring.profiles.active
            - name: DB_URL
              valueFrom:
                configMapKeyRef:
                  name: abc-configmap
                  key: db.url
            - name: DB_USERNAME
              valueFrom:
                configMapKeyRef:
                  name: abc-configmap
                  key: db.username
            - name: BAD_IMPORT_PATH
              valueFrom:
                configMapKeyRef:
                  name: abc-configmap
                  key: bad.import.path
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: abc-secret
                  key: db.password
          ports:
            - containerPort: 8080
              protocol: TCP
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: bad-import-file
              mountPath: /nfs/abc
      dnsPolicy: ClusterFirst
      securityContext:
        runAsGroup: 44337
        runAsNonRoot: true
        supplementalGroups:
          - 44337

the PV request is as follows

apiVersion: v1
kind: PersistentVolume
metadata:
  name: abc-tuc-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: classic-nfs
  mountOptions:
    - hard
    - nfsvers=3
  nfs:
    path: /tm03v06_vol3014
    server: tm03v06cl02.jit.abc.com
    readOnly: false

Now the openshift user has id

sh-4.2$ id
        uid=1031290500(1031290500) gid=44337(technical) groups=44337(technical),1031290500

RECENT UPDATE

Just to be clear with the problem, Below I have two commands from the same pod terminal,

sh-4.2$ cd /nfs/
sh-4.2$ ls -la (The first command I tried immediately after pod creation.)
total 8
drwxr-xr-x.  1 root  root    29 Nov 29 08:20 .
drwxr-xr-x.  1 root  root    50 Nov 30 08:19 ..
drwxrwx---. 14 technical technical 8192 Nov 28 19:06 abc
sh-4.2$ ls -la(few seconds later on the same pod terminal)
ls: cannot access abc: Permission denied
total 0
drwxr-xr-x. 1 root root 29 Nov 29 08:20 .
drwxr-xr-x. 1 root root 50 Nov 30 08:19 ..
d?????????? ? ?    ?     ?            ? abc

So the problem is that I see these question marks(???) on the mount point. The mounting is working correctly but I cannot access this /nfs/abc directory and I see this ????? for some reason

UPDATE

sh-4.2$ ls -la /nfs/abc/
ls: cannot open directory /nfs/abc/: Stale file handle
sh-4.2$ ls -la /nfs/abc/ (after few seconds on the same pod terminal)
ls: cannot access /nfs/abc/: Permission denied

Could this STALE FILE HANDLE be the reason for this issue?

Answer 1

TL;DR

You can use the anyuid security context to run the pod to avoid having OpenShift assign an arbitrary UID, and set the permissions on the volume to the known UID of the user.

---

OpenShift will override the user ID the image itself may specify that it should run as:

The user ID isn't actually entirely random, but is an assigned user ID which is unique to your project. In fact, your project is assigned a range of user IDs that applications can be run as. The set of user IDs will not overlap with other projects. You can see what range is assigned to a project by running oc describe on the project.

The purpose of assigning each project a distinct range of user IDs is so that in a multitenant environment, applications from different projects never run as the same user ID. When using persistent storage, any files created by applications will also have different ownership in the file system.

... this is a blessing and a curse, when using shared persistent volume claims for example (e.g. PVC's mounted in ReadWriteMany with multiple pods that read / write data - files created by one pod won't be accessible by the other pod because of the incorrect file ownership and permissions).

One way to get around this issue is using the anyuid security context which "provides all features of the restricted SCC, but allows users to run with any UID and any GID".

When using the anyuid security context, we know the user and group ID's the pod(s) are going to run as, and we can set the permissions on the shared volume in advance. For example, where all pods run with the restricted security context by default:

When running the pod with the anyuid security context, OpenShift doesn't assign an arbitrary UID from the range of UID's allocated for the namespace:

This is just for example, but an image that is built with a non-root user with a fixed UID and GID (e.g. 1000:1000) would run in OpenShift as that user, files would be created with the ownership of that user (e.g. 1000:1000), permissions can be set on the PVC to the known UID and GID of the user set to run the service. For example, we can create a new PVC:

cat <<EOF |kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: k8s
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 8Gi
  storageClassName: portworx-shared-sc
EOF

... then mount it in a pod:

kubectl run -i --rm --tty ansible --image=lazybit/ansible:v4.0.0 --restart=Never -n k8s --overrides='
{
  "apiVersion": "v1",
  "kind": "Pod",
  "spec": {
    "serviceAccountName": "default",
    "containers": [
      {
        "name": "nginx",
        "imagePullPolicy": "Always",
        "image": "lazybit/ansible:v4.0.0",
        "command": ["ash"],
        "stdin": true,
        "stdinOnce": true,
        "tty": true,
        "env": [
          {
            "name": "POD_NAME",
            "valueFrom": {
              "fieldRef": {
                "apiVersion": "v1",
                "fieldPath": "metadata.name"
              }
            }
          }
        ],
        "volumeMounts": [
          {
            "mountPath": "/data",
            "name": "data"
          }
        ]
      }
    ],
    "volumes": [
      {
        "name": "data",
        "persistentVolumeClaim": {
          "claimName": "data"
        }
      }
    ]
  }
}'

... and create files in the PVC as the USER set in the Dockerfile.