After upgrade to docker 18.09. all containers are invalid because of devicemapper error

Question

Today I upgraded docker from 18.06 to 18.09 in a debian stretch host. After the upgrade all our images and container were "gone", i.e. docker <command> won't show them anymore but the data in /var/lib/docker wasn't gone. dockerd -D unveiled:

Cannot load container XXX because it was created with another graph driver

So the first thing I did was downgrade docker to the last version to check if that fixes it. It worked kind of. The systemd unit file which worked before failed to start the docker service. Turns out I have to explicitly pass -s devicemapper to dockerd now and I don't know why because apparently the upgrade to 18.09 did put /var/lib/docker/overlay2 which I simply had to remove to make 18.06 startup as before.

So after I confirmed that this is fixed with the last docker-version, I upgraded again to 18.09 and tried if passing -s devicemapper explicitly does the trick here. But with 18.09 the daemon won't even start at all and complains:

Failed to GetDriver graph                     driver=devicemapper error="graphdriver plugins are only supported with experimental mode" home-dir=/var/lib/docker

I can't find anything about devicemapper being deprecated or something like that, so what were I supposed to do before the upgrade? Is there any general advice/best practice to avoid issues like that?

Answer 1

In case it is of help to anyone. I had to nuke the 18.09 release that was pulled in through the update channel for my mint installation:

sudo apt-get autoremove docker-ce 
sudo rm -rf /etc/docker 
sudo rm -rf /var/lib/docker
sudo apt-get autoremove --purge docker-ce

Then reinstall before getting a working 18.09 docker install back:

sudo apt-get install docker-ce

This moved my storage driver to Storage Driver: overlay2 when it was previously devicemapper, which seems to be the root of the issue.

There's no doubt a solution to this via /etc/docker/daemon.json settings, but I haven't got time to figure out what that is unfortunately.

Clearly, you'll lose all your local Docker images if you nuke /var/lib/docker, so make sure you want to do that before cut-n-pasting.

And yes, the original question should probably have been posted to SU and not SO...