Mongodb community operator kubernetes with aws eks and efs as storage

Question

I am implementing mongodb stateful using it's community operator on AWS EKS, and for persistent storage I am using EFS.

below is my yaml file, where I have mentioned my custom pvc, which is an efs.

---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: example-mongodb
spec:

  statefulSet:
    spec:
      volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            storageClassName: efs-sc
  members: 2
  type: ReplicaSet
  version: "6.0.5"
  security:
    authentication:
      modes: ["SCRAM"]
  users:

    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
        - name: readWrite
          db: admin



      scramCredentialsSecretName: my-scram
  additionalMongodConfig:
    storage.wiredTiger.engineConfig.journalCompressor: zlib

now when I am implementing it, it is creating required pv and pvc dynamically, and I can see that it is writing somoe file in the efs too, below is snapshot for the same.

but the pod is getting into crashloopbackoff, I am putting the logs of the pod below.

{"t":{"$date":"2024-01-11T08:55:38.344+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"6.0.5","gitVersion":"c9a99c120371d4d4c52cbb15dac34a36ce8d3b1d","openSSLVersion":"OpenSSL 3.0.2 15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2204","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2024-01-11T08:55:38.344+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"22.04"}}}
{"t":{"$date":"2024-01-11T08:55:38.344+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/data/automation-mongod.conf","net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"example-mongodb"},"security":{"authorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data"}}}}
{"t":{"$date":"2024-01-11T08:55:38.385+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=256M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],"}}
{"t":{"$date":"2024-01-11T08:55:39.170+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":1,"message":{"ts_sec":1704963339,"ts_usec":170490,"thread":"1:0x7fb648844cc0","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:812:/data/WiredTiger.wt: handle-open: open","error_str":"Operation not permitted","error_code":1}}}
{"t":{"$date":"2024-01-11T08:55:39.238+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":17,"message":{"ts_sec":1704963339,"ts_usec":238564,"thread":"1:0x7fb648844cc0","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:812:/data/WiredTiger.wt: handle-open: open","error_str":"File exists","error_code":17}}}
{"t":{"$date":"2024-01-11T08:55:39.248+00:00"},"s":"I",  "c":"WT",       "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1704963339,"ts_usec":248704,"thread":"1:0x7fb648844cc0","session_name":"connection","category":"WT_VERB_BLOCK","category_id":3,"verbose_level":"NOTICE","verbose_level_id":-1,"msg":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1"}}}
{"t":{"$date":"2024-01-11T08:55:39.258+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":1,"message":{"ts_sec":1704963339,"ts_usec":258672,"thread":"1:0x7fb648844cc0","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:812:/data/WiredTiger.wt: handle-open: open","error_str":"Operation not permitted","error_code":1}}}
{"t":{"$date":"2024-01-11T08:55:39.318+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":17,"message":{"ts_sec":1704963339,"ts_usec":318557,"thread":"1:0x7fb648844cc0","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:812:/data/WiredTiger.wt: handle-open: open","error_str":"File exists","error_code":17}}}
{"t":{"$date":"2024-01-11T08:55:39.329+00:00"},"s":"I",  "c":"WT",       "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1704963339,"ts_usec":329772,"thread":"1:0x7fb648844cc0","session_name":"connection","category":"WT_VERB_BLOCK","category_id":3,"verbose_level":"NOTICE","verbose_level_id":-1,"msg":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.2"}}}
{"t":{"$date":"2024-01-11T08:55:39.339+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":1,"message":{"ts_sec":1704963339,"ts_usec":339405,"thread":"1:0x7fb648844cc0","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:812:/data/WiredTiger.wt: handle-open: open","error_str":"Operation not permitted","error_code":1}}}
{"t":{"$date":"2024-01-11T08:55:39.350+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2024-01-11T08:55:39.350+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"1: Operation not permitted"}}
{"t":{"$date":"2024-01-11T08:55:39.350+00:00"},"s":"F",  "c":"ASSERT",   "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":708}}
{"t":{"$date":"2024-01-11T08:55:39.350+00:00"},"s":"F",  "c":"ASSERT",   "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

When I am doing same without specifying my custom pvc (in that case it is using ebs of the nodegroup) it is working fine but when I am using my custom pvc it is throwing me that error.

Thanks for considering.

Answer 1

The EFS volumes mounted in the Mongod container appear to lack the correct file system permissions. The issue was resolved by aligning the uid and gid parameters with the security context in the StatefulSet and adjusting the directory permissions on the EFS storage claim.

A similar issue was reported which helped to figure the issue. https://github.com/bitnami/charts/issues/8350

Security context in StatefulSet:

securityContext:
  fsGroup: 1001
  runAsGroup: 1001
  runAsUser: 1001

Full MongoDBCommunity resource with the above security context applied:

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: mongo
spec:
  members: 2
  type: ReplicaSet
  version: "6.0.5"
  security:
    authentication:
      modes: ["SCRAM"]
  
  users:
    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: root
          db: admin
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram
  additionalMongodConfig:
    storage.wiredTiger.engineConfig.journalCompressor: zlib
    storage.wiredTiger.engineConfig.cacheSizeGB: 1
  statefulSet:
    spec:      
      volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
        
      template:
        spec:
          securityContext:
            fsGroup: 1001
            runAsGroup: 1001
            runAsUser: 1001
          containers:            
            - name: mongod              
              resources:
                limits:
                  cpu: "3900m"
                  memory: 14Gi
                requests:
                  cpu: 3
                  memory: 12Gi

EFS StorageClass:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-xxxxx
  directoryPerms: "777"
  uid: "1001"
  gid: "1001"
  basePath: "/dynamic_provisioning"