NFS Persistant Volume monitoring in prometheus

Question

I have a Persistent Volume of type NFS with a source server running locally. This NFS server is using a Persistent Volume Claim that lives on GCP. Everything is running fine, however, I cannot monitor the PVC on Prometheus for some reason. It's just not showing up.

Question: How can I monitor the usage of such disks? Do I need to install some drivers or configure custom scraping?

Here are the PromQL queries I tried:

kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_available_bytes
node_filesystem_size_bytes

Note: I'm able to see the volume here, but I can only see it's capacity, not the usage.

kube_persistentvolume_capacity_bytes

Here's the config file for the NFS server:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nfs-server
spec:
  replicas: 1
  selector:
    matchLabels:
      role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      containers:
      - name: nfs-server
        image: gcr.io/google_containers/volume-nfs:0.8
        ports:
          - name: nfs
            containerPort: 2049
          - name: mountd
            containerPort: 20048
          - name: rpcbind
            containerPort: 111
        securityContext:
          privileged: true
        volumeMounts:
          - mountPath: /exports
            name: nfs-pvc
      volumes:
        - name: nfs-pvc
          gcePersistentDisk:
            pdName: nfs-disk
            fsType: ext4

Here's the PV config:

Status:          Bound
Claim:           default/nfs-pvc
Reclaim Policy:  Retain
Access Modes:    RWX
VolumeMode:      Filesystem
Capacity:        1000Gi
Node Affinity:   <none>
Message:
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    nfs-server.default.svc.cluster.local
    Path:      /
    ReadOnly:  false

Here's the PVC config:

Name:          nfs-pvc
Namespace:     default
StorageClass:
Status:        Bound
Volume:        nfs-pv
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1000Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Mounted By:    ...

• Kubernetes Version: 1.16.15-gke.500
• Prometheus Version: 2.21.0
• Prometheus Helm Chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus

Answer 1

Volume Exporter to the rescue, it can easily done in the following way.

Volume exporter is created specifically for these kind of needs, where in node exporter may not be useful (It basically fills the void that exists)

      - name: volume-exporter
      image:  mnadeem/volume_exporter
      imagePullPolicy: "Always"
      args:
        - --volume-dir=prometheus:/prometheus
        - --volume-dir=appLog:/app/log
      ports:
      - name: metrics-volume
        containerPort: 9888
      volumeMounts:
      - name: prometheus-data
        mountPath: /prometheus
        readOnly: true
      - name: log-data-volume
        mountPath: /app/log
        readOnly: true

Exported metrics

            # HELP volume_bytes_free Free size of the volume/disk
            # TYPE volume_bytes_free gauge
            volume_bytes_free{volume_name="bin",volume_path="/bin"} 4.341569536e+10
            volume_bytes_free{volume_name="etc",volume_path="/etc"} 4.341569536e+10
            # HELP volume_bytes_total Total size of the volume/disk
            # TYPE volume_bytes_total gauge
            volume_bytes_total{volume_name="bin",volume_path="/bin"} 6.391887872e+10
            volume_bytes_total{volume_name="etc",volume_path="/etc"} 6.391887872e+10
            # HELP volume_bytes_used Used size of volume/disk
            # TYPE volume_bytes_used gauge
            volume_bytes_used{volume_name="bin",volume_path="/bin"} 2.050318336e+10
            volume_bytes_used{volume_name="etc",volume_path="/etc"} 2.050318336e+10
            # HELP volume_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which volume_exporter was built.
            # TYPE volume_exporter_build_info gauge
            volume_exporter_build_info{branch="",goversion="go1.15",revision="",version=""} 1
            # HELP volume_percentage_used Percentage of volume/disk Utilization
            # TYPE volume_percentage_used gauge
            volume_percentage_used{volume_name="bin",volume_path="/bin"} 32.07688208958619
            volume_percentage_used{volume_name="etc",volume_path="/etc"} 32.07688208958619

Disclaimer : I am the owner