Copy out file tree from Docker layers on Mac OS X

Question

I made some local changes to a Docker container and commited them, and am now trying to replicate them when future instances are built. However, checking which specific changes were made is proving difficult. I know the layers which contain changes, but existing tools (e.g. wagoodman/dive) do not detect this type of change well.

The layers are stored in /var/lib/docker/overlay2/, but I am on Mac OSX, where that does not exist as a literal path, and must be accessed by connecting to the virtual filesystem in a container. This is possible through a workaround, but this method opens the filesystem read-only and it lacks most tools:

docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh

I have used this to verify that the layers look roughly as I expect. To look in more detail, I want to extract the file trees to the main filesystem where my tools are present via docker cp. The nsenter invocation is running as container festive_lehmann, and in that session I have the shell open at /var/lib/docker/overlay2/be90015c1ff3efd16d145dc079edc78b5ee59f65b947b68c06b0c4885bbf9d84/diff/. However, docker cp in another session produces this error:

$ docker cp festive_lehmann:/var/lib/docker/overlay2/be90015c1ff3efd16d145dc079edc78b5ee59f65b947b68c06b0c4885bbf9d84/diff/. ./diff/
Error: No such container:path: festive_lehmann:/var/lib/docker/overlay2/be90015c1ff3efd16d145dc079edc78b5ee59f65b947b68c06b0c4885bbf9d84/diff

I checked that this is not specific to copying a directory:

$ docker cp festive_lehmann:/var/lib/docker/overlay2/be90015c1ff3efd16d145dc079edc78b5ee59f65b947b68c06b0c4885bbf9d84/link .
Error: No such container:path: festive_lehmann:/var/lib/docker/overlay2/be90015c1ff3efd16d145dc079edc78b5ee59f65b947b68c06b0c4885bbf9d84/link

Since I literally have a shell open at that path, I presume this is some kind of problem with the container being inaccessible to normal operations, the same thing that makes it necessary to use the arcane nsenter tricks to connect to it. But I do not actually understand those tricks, and the relevant flags (-i, -t, --privileged, --pid) are not accepted for docker cp. Is there a correct invocation, or a workaround?

Answer 1

The docker cp command is copying from the container filesystem, inside the container namespace. The nsenter command is changing your namespace so you no longer see the container filesystem in the commands you're running from that shell.

My suggestion if you want to view these changes is either to use the dive tool, or you can do it by hand with a docker save to save the image contents to a tar file that contains the tar+gzip layer blobs. Dissecting that structure isn't too difficult, you'll see a json file at the top directing you to the names of each of the layers, and the files can be extracted with a tar command (e.g. tar -xzvf $blob_name -C $extract_dir).

The other option is to launch a command like:

docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i \
  tar -cf /var/lib/docker/overlay2/be90015c1ff3efd16d145dc079edc78b5ee59f65b947b68c06b0c4885bbf9d84/diff \
  | tar -xvf -