Postgres AWS EBS volume doesn't persist when updating service

Question

I deploy a service on a standard Docker for AWS stack (using this template). I deploy using docker stack deploy -c docker-compose.yml pos with this compose file:

version: "3.2"
services:
  postgres_vanilla:
    image: postgres
    volumes:
      - db-data:/var/lib/postgresql
volumes:
  db-data:
    driver: "cloudstor:aws"
    driver_opts:
      size: "6"
      ebstype: "gp2"
      backing: "relocatable"

I then change some data in the db and force an update of the service with docker service update --force pos_postgres_vanilla

Problem is that the data I change doesn't persist after the update. I've noticed that postgres initdb script runs every time I update, so I assume it's related.

Is there something i'm doing wrong?

Answer 1

Issue was that cloudstor:aws creates the volume with a lost+found under it, so when postgres starts it finds that the data directory isn't empty and complains about it. To fix that I changed the volume to be mounted one directory above the data directory, at /var/lib/postgresql, but that caused postgres to not find the PGVERSION file, which in turn caused it to run initdb every time the container starts (https://github.com/docker-library/postgres/blob/master/11/docker-entrypoint.sh#L57).

So to work around it, instead of changing the volume to be mounted one directory above the data directory, I changed the data directory to be one level below the volume mount by overriding environment variable PGDATA (to something like /var/lib/postgresql/data/db/).