How to upgrade a Helm chart or image version of an application deployment in kubernetes without losing data in associated PV?

Question

I am attempting to update a MinIO deployment in my Kubernetes cluster, and I am having some trouble in keeping access to any data that was uploaded to the MinIO deployment before upgrading the Helm chart or the image version. This is specifically about MinIO, but this problem (and potential solution) could be applied to any application deployment that requires storage and PV's/PVC's.

So I want to upgrade my MinIO deployment image + helm chart to the next one up but whenever I do this, I lose all my data, any buckets created, any files uploaded to buckets, any access policies, literally everything. I have my storage connected AWS EFS.

I found the 1st issue which was that we were using storageClasses and performing dynamic provision of PV's and PVC's, so when a deployment was changed or updated (not changing Helm chart values, only specifically changing the Helm chart version or MinIO image version) a new access point was created and a new PV and PVC was also created, therefore, there was none of the existing data in these new objects. So I did some googling and found that I can statically provision an EFS access point and manually create a PV and PVC. My PV can then refer to that access point ID in my PV using the spec.volumeHandle definition, something like this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio
spec:
  capacity:
    storage: 50Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-<numbers>::fsap-<numbers>

On top of this, I also needed to add a spec.volumeName definition to my PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  volumeName: minio

With this configuration, it looks like I am able to use the same PV and PVC and new volumes are NOT being provisioned. "Great!" I thought, so I tested an upgrade of the Helm chart version, but all my data still disappears, even though the same PV and PVC are used in the upgraded deployment.

I'm really not sure what I'm doing wrong now as there are definitely no new volumes being provisioned and I am definitely using the existing volume.

How can I upgrade my MinIO deployment without losing the data?? I thought it was as simple as using the same PV but I must be missing something.

Answer 1

You should almost never manually create PersistentVolumes, and usually not PersistentVolumeClaims either.

If your application uses a StatefulSet, one of the parts of that is a list of volumeClaimTemplates. This instructs the cluster to create a PersistentVolumeClaim for each replica, and then the cluster will also automatically create a PersistentVolume and allocate storage.

apiVersion: apps/v1
kind: StatefulSet
spec:
  volumeClaimTemplates:
    - name: minio
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 50Gi
  template:
    spec:
      containers:
        - name: minio
          image: bitnami/minio
          volumeMounts:
            - name: minio
              mountPath: /bitnami/minio/data

There are two more reasons a StatefulSet is helpful here. First, deleting a StatefulSet doesn't normally delete the corresponding PersistentVolumeClaim, so the data volume remains in the cluster even on upgrade (or deletion and recration). Second, the name of the PersistentVolumeClaim the StatefulSet creates is predictable, so you should see something named minio-ssname-0 where ssname is the StatefulSet name; that means that, is the StatefulSet is deleted and recreated, it can find the existing PVC.

In an AWS/EKS context, the cluster will normally provision an EBS volume with at least the specified minimum size for each PersistentVolumeClaim. You should not need to manually create an EFS volume or know the AWS IDs of any of the objects that get created.

Your question focuses a lot on the mechanics of PersistentVolume(Claims), but your high-level symptom is that you lose data on any update. It's also worth verifying that you're mounting the volume on the correct container path. The example above uses the bitnami/minio image and its documented container path. If you get the container path wrong, you'll get exactly the symptoms you describe: things apparently work, but data is stored in the container-temporary filesystem, so as soon as anything deletes a Pod (like a routine upgrade) the data will vanish.