Mounting azure files nfs share to pod

Question

attempting to mount an azure files nfs share to a kubernetes pod but getting this:

MountVolume.MountDevice failed for volume "#azurestorage#fileshare2#azurestorage-volume#ephemeral--azure-storage-poc" : rpc error: code = Internal desc = volume(#azurestorage#fileshare2#azurestorage-volume#ephemeral--azure-storage-poc) mount //.file.core.windows.net/fileshare2 on /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com//globalmount failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t cifs -o dir_mode=0777,actimeo=30,mfsymlinks,nosharesock,file_mode=0777, //stgactdbanfs.file.core.windows.net/fileshare2 /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com//globalmount Output: mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg) Please refer to http://aka.ms/filemounterror for possible causes and solutions for mount errors.

The key issue being mount error 13. Per the documentation:

• I've confirmed that it has a private endpoint
• I've confirmed that there is a virtual network link between the virtual network hosting the aks cluster and the private dns zone containing the private endpoint
• I've also explicity enabled traffic from selected networks and opened up traffic from the subnet within the vnet for the cluster into the storage account

I've also noticed per the documentation that the error message is referencing .file.core.windows.net as opposed to .file.core.windows.net, suggesting its not resolving against the private endpoint

However i have the private endpoint specified in the storage class

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurestoragepoc4
provisioner: file.csi.azure.com 
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=0
  - gid=0
  - mfsymlinks
  - cache=strict
  - actimeo=30
parameters:
  skuName: Premium_LRS
  location: eastus2
  resourceGroup: <resource-group>
  accountName: <storageaccout>.privatelink.file.core.windows.net
  shareName: fileshare2
  protocol: nfs

and the pv:

apiVersion: "v1"
kind: "PersistentVolume"
metadata:
  name: "pv0002"
spec:
  capacity:
    storage: "100Mi"
  accessModes:
  - "ReadWriteMany"
  storageClassName: azurestoragepoc4
  azureFile:
    accountName: <storageaccount>.privatelink.file.core.windows.net
    secretName: accountkey
    shareName: fileshare2
    readOnly: false
    claimRef:
      name: claim4

The secretName value is pulled in via Vault

Not sure what im missing here, i've confirmed the value in vault is correct, connectvity is open. Am i missing something specific to the aks vmss? I've read from other articles that this error code is often associated with missing dependencies like nfs-common-utils

any insight?

Answer 1

The error you're encountering, mount error(13): Permission denied, typically indicates a problem with authentication or authorization when trying to mount the NFS share. Based on the detailed description of your setup and troubleshooting steps, it seems like you've covered many of the common issues. However, a few areas could still be problematic or worth rechecking. The error message suggests a potential DNS resolution issue, as it's trying to resolve .file.core.windows.net instead of using the private endpoint you've specified. This might indicate a problem with the DNS setup in your AKS cluster. The mount options you're using are for CIFS/SMB (-t cifs), but you've mentioned wanting to use NFS. NFS and CIFS are different protocols, and the mount options for each are not interchangeable.

Azure Files is a managed file share service in Azure. Initially, the service only supported CIFS/SMB as a protocol to mount/access the files. CIFS/SMB is similar to NFS, but it’s a different protocol. To make Azure Files more friendly to Linux users, recently a preview for Azure Files exposing an NFS endpoint was announced.

In the below example, I’ll set up Azure Files with NFS using a private endpoint.

First, we need to register the preview feature:

az feature register --name AllowNfsFileShares --namespace Microsoft.Storage
az provider register --namespace Microsoft.Storage

It takes a while for this to register. To get the status, use the following.

az feature show --name AllowNfsFileShares --namespace Microsoft.Storage --query properties.state

When we’re ready, we can create the network we need for this work:

RGNAME=nfsaks
VNETNAME=nfsaks
az group create -n $RGNAME -l westus2
az network vnet create -g $RGNAME -n $VNETNAME \
  --address-prefixes 10.0.0.0/16 --subnet-name aks \
  --subnet-prefixes 10.0.0.0/24
az network vnet subnet create -g $RGNAME --vnet-name $VNETNAME \
  -n NFS --address-prefixes 10.0.1.0/24 

Next, let’s create the storage account, turn of HTTPS only, and create the share:

STACC=nfnfsaks
az storage account create \
  --name $STACC \
  --resource-group $RGNAME \
  --location westus2 \
  --sku Premium_LRS \
  --kind FileStorage
az storage account update --https-only false \
  --name $STACC --resource-group $RGNAME
az storage share-rm create \
  --storage-account $STACC \
  --enabled-protocol NFS \
  --root-squash RootSquash \
  --name "akstest" \
  --quota 100

Then next, we need to set up a private endpoint for this.

SUBNETID=`az network vnet subnet show \
  --resource-group $RGNAME \
  --vnet-name $VNETNAME \
  --name NFS \
  --query "id" -o tsv `
STACCID=`az storage account show \
  --resource-group $RGNAME \
  --name $STACC \
  --query "id" -o tsv `
az network vnet subnet update \
  --ids $SUBNETID\
  --disable-private-endpoint-network-policies 
ENDPOINT=`az network private-endpoint create \
  --resource-group $RGNAME \
  --name "$STACC-PrivateEndpoint" \
  --location westus2 \
  --subnet $SUBNETID \
  --private-connection-resource-id $STACCID\
  --group-id "file" \
  --connection-name "$STACC-Connection" \
  --query "id" -o tsv `

Once we have the private endpoint created, we also need to create a DNS zone and set up an A-record for our storage account:

DNSZONENAME="privatelink.file.core.windows.net"
VNETID=`az network vnet show \
  --resource-group $RGNAME \
  --name $VNETNAME \
  --query "id" -o tsv`
dnsZone=`az network private-dns zone create \
  --resource-group $RGNAME \
  --name $DNSZONENAME \
  --query "id" -o tsv`
az network private-dns link vnet create \
  --resource-group $RGNAME \
  --zone-name $DNSZONENAME \
  --name "$VNETNAME-DnsLink" \
  --virtual-network $VNETID \
  --registration-enabled false 

ENDPOINTNIC=`az network private-endpoint show \
  --ids $ENDPOINT \
  --query "networkInterfaces[0].id" -o tsv `
ENDPOINTIP=`az network nic show \
  --ids $ENDPOINTNIC \
  --query "ipConfigurations[0].privateIpAddress" -o tsv `

az network private-dns record-set a create \
        --resource-group $RGNAME \
        --zone-name $DNSZONENAME \
        --name $STACC 
az network private-dns record-set a add-record \
        --resource-group $RGNAME \
        --zone-name $DNSZONENAME  \
        --record-set-name $STACC \
        --ipv4-address $ENDPOINTIP 

Mounting NFS in AKS

Let’s first create a new cluster in the AKS subnet we pre-created before:

AKSSUBNETID=`az network vnet subnet show \
  --resource-group $RGNAME \
  --vnet-name $VNETNAME \
  --name AKS\
  --query "id" -o tsv `
az aks create -g $RGNAME -n nfstest \
 --vnet-subnet-id $AKSSUBNETID \
 --service-cidr 10.1.0.0/16 \
 --dns-service-ip 10.1.0.10 \
 --network-plugin kubenet
az aks get-credentials -g $RGNAME -n nfstest

Once we have the cluster created, we can go ahead and create a pod that mounts our NFS share.

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: myapp
    image: busybox
    command: ["/bin/sh", "-ec", "sleep 1000"]
    volumeMounts:
      - name: nfs
        mountPath: /var/nfs
  volumes:
  - name: nfs
    nfs:
      server: nfnfsaks.file.core.windows.net
      path: "/nfnfsaks/akstest"

We can create this pod using kubectl create -f pod-with-nfs.yaml.

Once we have the pod, we can exec into it, and see what’s available on the file share.

And that’s how you can mount Azure Files over NFS in an Azure Kubernetes cluster.

References:

• Errors when mounting an Azure file share
• Mount error(13): Permission denied