I am trying to pull and save a few docker images in parallel (they are quite big and parallelism can save a lot of time - it is done within a Python script activating docker pull and then save within each thread). However, it fails all the time with the message like this:
Client Error: Not Found ("open /var/lib/docker/overlay2/41693d132695cd5ada8cf37f210d5b70bc1bac1b2cedfa5a4f352efa5ff00fc6/merged/some_file_name: no such file or directory")
the specific file on which it complains ('no such file or directory') varies. In /var/log/messages (even after adding the debug flag to docker daemon options) I can't see anything valuable. e.g.
level=error msg="Handler for GET /v1.35/images/xxx/xxx:xxx/get returned error: open /var/lib/docker/overlay2/41693d132695cd5ada8cf37f210d5b70bc1bac1b2cedfa5a4f352efa5ff00fc6/merged/opt/external/postgresql-42.2.5/postgresql-42.2.5.ja r: no such file or directory"
Important (probably) note: - the images share many layers in common as they are built based on the same parent images (is this the reason for the collision in overlay FS?). Running the same sequentually (number of parallel threads set to 1) works perfectly
OS: centos 7.9 Docker: Server Version: 1.13.1 Storage Driver: overlay2 Backing Filesystem: xfs