Redis db permission issue while running GitLab

Question

I am trying to setup GitLab, Redis and PostgreSQL containers in Kubernetes using Gluster for persistence. GlusterFS nodes are setup on machines (CentOS) external to Kubernetes cluster (running on RancherOS host). Issue is that when GitLab tries starting up, the login page doesn't load. It's a fresh setup and not something that stopped working now.

root@gitlab-2797053212-ph4j8:/var/log/gitlab/gitlab# tail -50 sidekiq.log
...
...
   2017-09-07T11:53:03.098Z 547 TID-1fdjck ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:84:in `process_one'
2017-09-07T11:53:03.098Z 547 TID-1fdjck ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:73:in `run'
2017-09-07T11:53:03.098Z 547 TID-1fdjck ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/util.rb:17:in `watchdog'
2017-09-07T11:53:03.098Z 547 TID-1fdjck ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/util.rb:26:in `block in safe_thread'
2017-09-07T11:53:03.099Z 547 TID-1fdf1k ERROR: Error fetching job: ERR Error running script (call to f_7b91ed9f4cba40689cea7172d1fd3e08b2efd8c9): @user_script:7: @user_script: 7: -MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:121:in `call'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/peek-redis-1.2.0/lib/peek/views/redis.rb:9:in `call'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-3.3.3/lib/redis.rb:2399:in `block in _eval'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'
...

So i checked for Redis container logs.

[root@node-a ~]# docker logs -f 67d44f585705
...
...
[1] 07 Sep 14:43:48.140 # Background saving error
[1] 07 Sep 14:43:54.048 * 1 changes in 900 seconds. Saving...
[1] 07 Sep 14:43:54.048 * Background saving started by pid 2437
[2437] 07 Sep 14:43:54.053 # Failed opening .rdb for saving: Permission denied
...

Checked online for this issue and then noticed the following permissions and owner details inside of Redis pod:

[root@node-a ~]# docker exec -it 67d44f585705 bash
groups: cannot find name for group ID 2000
root@redis-2138096053-0mlx4:/# ls -ld /var/lib/redis/
drwxr-sr-x 12 1000 1000 8192 Sep  7 11:51 /var/lib/redis/
root@redis-2138096053-0mlx4:/#
root@redis-2138096053-0mlx4:/# ls -l /var/lib/redis/
total 22
drwxr-sr-x 2  1000  1000     6 Sep  6 10:37 backups
drwxr-sr-x 2  1000  1000     6 Sep  6 10:37 builds
drwxr-sr-x 2 redis redis     6 Sep  6 10:14 data
-rw-r--r-- 1 redis redis 13050 Sep  7 11:51 dump.rdb
-rwxr-xr-x 1 redis redis    21 Sep  5 11:00 index.html
drwxrws--- 2  1000  1000     6 Sep  6 10:37 repositories
drwxr-sr-x 5  1000  1000    55 Sep  6 10:37 shared
drwxr-sr-x 2 root  root   8192 Sep  6 10:37 ssh
drwxr-sr-x 3 redis redis    70 Sep  7 10:20 tmp
drwx--S--- 2  1000  1000     6 Sep  6 10:37 uploads
root@redis-2138096053-0mlx4:/#
root@redis-2138096053-0mlx4:/# grep 1000 /etc/passwd
root@redis-2138096053-0mlx4:/#

Ran following and all looked fine.

root@redis-2138096053-0mlx4:/# chown redis:redis -R /var/lib/redis/

However, when i deleted and ran the GitLab deployment YAML again, the permissions inside the Redis container again got skewed up. I am not sure whether Gluster is messing up with the Redis file/folders permissions. Can't think of any other reason right now.

One thing i would like to highlight is that all the three containers are using the same PVC

- name: gluster-vol1
  persistentVolumeClaim:
    claimName: gluster-dyn-pvc

Above is common for all three. What differs is shown below:

a) postgresql-deployment.yaml

volumeMounts:
- name: gluster-vol1
  mountPath: /var/lib/postgresql

b) redisio-deployment.yaml

volumeMounts:
- name: gluster-vol1
  mountPath: /var/lib/redis

c) gitlab-deployment.yaml

volumeMounts:
- name: gluster-vol1
  mountPath: /home/git/data

Any suggestion?

Answer 1

With the following steps, i was able to resolve the 'Permission denied' issue for Redis:

1. Created separate volumes for PostegreSQL, Redis and GitLab in GlusterFS.
2. Created separate Storage Class for all three.
3. Created PersistentVolumeClaim (PVC) for them and mapped /var/lib/postgresql, /var/lib/redis and /home/git/data to their respective PVCs.

Earlier, all three paths mentioned above were pointing to the same volume in GlusterFS. Somehow, it seems, they were causing issue for Redis.