Local Home requires POSIX when deploying Bitbucket on AKS using azurefile csi

Question

I am deploying Bitbucket to AKS cluster following the Atlassian installation guide: Install Atlassian DC Helm Charts and using azurefile-csi to dynamically create both PV and PVC for local home.

After deploying the helm chart, the pod with Bitbucket starts but never gets to a ready state due to the following error:

2024-01-08 17:52:21,922 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started


org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshHealthcheck' defined in URL [jar:file:/opt/atlassian/bitbucket/app/WEB-INF/lib/bitbucket-git-mesh-8.9.8.jar!/com/atlassian/stash/internal/scm/git/mesh/MeshHealthcheck.class]:

 Unsatisfied dependency expressed through constructor parameter 0; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshClient' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'meshClient' parameter 2; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'meshRouter' parameter 5; 

 nested exception is org.springframework.beans.factory.BeanCreationException:

 Error creating bean with name 'sidecarManager': 

 Invocation of init method failed; 

 nested exception is java.nio.file.FileSystemException: 

 /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem:

 Operation not permitted

 ...

 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)

 ... 5 common frames omitted

Caused by: java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)

 at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)

 at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)

 at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:254)

 at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:276)

 at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.write(KeyUtils.java:382)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.writePublicKey(KeyUtils.java:260)

 at com.atlassian.stash.internal.scm.git.mesh.MeshLauncher.start(MeshLauncher.java:121)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:272)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.internalStart(DefaultSidecarManager.java:223)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:178)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 ... 5 common frames omitted

When checking the stack trace I realized that the error:

Caused by: java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

is being thrown when executing:

at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)
at com.atlassian.bitbucket.mesh.util.KeyUtils.write(KeyUtils.java:382)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.writePublicKey(KeyUtils.java:260)

 at com.atlassian.stash.internal.scm.git.mesh.MeshLauncher.start(MeshLauncher.java:121)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:272)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.internalStart(DefaultSidecarManager.java:223)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:178)

For Local Home I am dynamically creating both pv and pvc using the built in storage class azurefile-csi from Dynamically create azure files.

For local home I am passing the follow values on values.yaml

localHome:

# Dynamic provisioning of local-home using the K8s Storage Classes
#
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic
# https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/
#
persistentVolumeClaim:

# -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically
# created for each pod based on the 'StorageClassName' supplied below.
#
create: true

# -- Specify the name of the 'StorageClass' that should be used for the local-home
# volume claim.
#
storageClassName: "azurefile-csi"

# -- Specifies the standard K8s resource requests and/or limits for the local-home
# volume claims.
#
resources:
requests:
storage: 1Gi

I also tried to create a claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-local-home
namespace: bitbucket
spec:
accessModes:
- ReadWriteMany
storageClassName: "azurefile-csi" 
resources:
requests:
storage: 1Gi

and pass it as:

localHome:

# Dynamic provisioning of local-home using the K8s Storage Classes
#
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic
# https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/
#
persistentVolumeClaim:

# -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically
# created for each pod based on the 'StorageClassName' supplied below.
#
create: false

...

# -- Static provisioning of local-home using K8s PVs and PVCs
#
# NOTE: Due to the ephemeral nature of pods this approach to provisioning volumes for
# pods is not recommended. Dynamic provisioning described above is the prescribed
# approach.
#
# When 'persistentVolumeClaim.create' is 'false', then this value can be used to define
# a standard K8s volume that will be used for the local-home volume(s). If not defined,
# then an 'emptyDir' volume is utilised. Having provisioned a 'PersistentVolume', specify
# the bound 'persistentVolumeClaim.claimName' for the 'customVolume' object.
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#static
#
customVolume:
 persistentVolumeClaim:
 claimName: "pvc-local-home"
 

Both resulted in the same error shown above.

Bitbucket has issues when trying to set POSIX in Azure Files as Azure files CSI is not compliant with POSIX, for that an NFS File Share has to be used.

When running the helm install using emptyDir for local home the issue does not happen.

I couldn't find any mention of POSIX in the section 5 Configure Persistent Storage of the installation guide.

Am I doing something wrong or a volume compliant with POSIX is needed in order to run Bitbucket properly on Kubernetes and that was not mentioned anywhere in the documentation?

Answer 1

Azure Files, lacks full POSIX compliance, you can switch to using Azure Disks or set up an NFS server. Azure Disks are more POSIX-compliant but can only be mounted to a single node, making them suitable for single-node setups while setting up an NFS server allows for multi-node access while maintaining POSIX compliance. Follow the guide, please modify the persistence section inside the values.yaml which you got from helm get values bitbucket -n . with the Azure disk details

persistence:
  localHome:
    enabled: true
    storageClassName: "your AzureDiskstorageclass"
    size: "10Gi"
  sharedHome:
    enabled: true
    storageClassName: "your AzureDiskstorageclass"
    size: "20Gi"

After that, deploy bitbucket.

helm install bitbucket atlassian-data-center/bitbucket --namespace default --values values.yaml 

kubectl get pods

You can also try setting up NFS and accordingly update the bitbucket's values.yaml file to reference to the NFS

localHome:
  customVolume:
    persistentVolumeClaim:
      claimName: bitbucket-nfs-pvc

and then redeploy bitbucket using helm upgrade <release-name> atlassian-data-center/bitbucket --namespace <namespace> --values values.yaml

Reference document: Bitbucket Setup Guide NFS for Bitbucket