Docker Registry: How do I get the list of digest of an image layers?

Question

I am trying to list the layers digest of an image (e.g.: ubuntu) that is loaded to a docker registry. I am basically looking for the equivalent of docker manifest inspect --verbose <image-name> for docker registry and which gives a similar output:

From the server that is hosting the registry, I am running the following command successfully to list the images inside the registry:

curl -k -X GET -u admin:root https://docker.registry.url/v2/_catalog | python -mjson.tool

I can also run the following command to see the tag of my image:

curl -k -X GET -u admin:root https://docker.registry.url/v2/ubuntu/tags/list | python -mjson.tool

But when I try to use the following command (as in here) to list the digests:

curl -k -v -s -X GET -u admin:root https://docker.registry.url/v2/ubuntu/manifests/latest -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' | python -mjson.tool

... I get the following two messages:

• "HTTP/1.1 404 Not Found"
• "errors: "code": "MANIFEST_UNKNOWN"

as per the following output:

* About to connect() to docker.registry.url port 443 (#0)
*   Trying <server-ip-address>...
* Connected to docker.registry.urll (<server-ip-address>) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=docker.registry.url
*       start date: Oct 22 13:15:18 2021 GMT
*       expire date: Oct 21 13:00:19 2031 GMT
*       common name: docker.registry.url
*       issuer: CN=docker.registry.url
* Server auth using Basic with user 'admin'
> GET /v2/ubuntu/manifests/latest HTTP/1.1
> Authorization: Basic YWRtaW46QWlyc3BhbjEyMw==
> User-Agent: curl/7.29.0
> Host: docker.registry.url
> Accept: application/vnd.docker.distribution.manifest.v2+json
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.21.3
< Date: Tue, 08 Mar 2022 15:00:36 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 187
< Connection: keep-alive
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
<
{ [data not shown]
* Connection #0 to host docker.registry.url left intact
{
    "errors": [
        {
            "code": "MANIFEST_UNKNOWN",
            "detail": {
                "Name": "ubuntu",
                "Revision": "sha256:9c152418e380c6e6dd7e19567bb6762b67e22b1d0612e4f5074bda6e6040c64a"
            },
            "message": "manifest unknown"
        }
    ]
}

1- Why am I getting the above error?

2- How do I get the list of manifests of an image from a docker registry?

Answer 1

Workaround:

Not sure if this answer is 100% related to the actual question, but I needed to delete images in my registry; using registry:2.8.1, followed this but even garbage collector didn't clear images for me so as a workaround I'm clearing images by not having them mounted as a volume - and just restarting the whole registry; and if I need to clear a different group of images independently, I can set up a second registry on a different port and also not use a mounted volume.

But in case it works for someone else here's some other commands:

1 Get Digest

docker inspect example.com:5000/myimage

there should be this section:

"RepoDigests": [
"my.docker.registry.com:5000/ubuntu@sha256:74a1b5f5c5d771cf3570fa0f050e0c827538b7fe1873bc88b85d56818df3f2bc"
]

2 run garbage collector

docker exec registry bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml

docker exec registry bin/registry garbage-collect --help /etc/docker/registry/config.yml

3 delete

curl -X DELETE -u username:password https://example.com:5000/v2/my_image/manifests/<digest> -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'

Answer 2

Given the error message, the registry understands the tag, and is trying to pull a specific digest but cannot find it. One of the most common causes of this is running a GC on the registry with a multi-platform image because of a longstanding bug with the GC design.

In the command:

curl -k -v -s -X GET -u admin:root \
  https://docker.registry.url/v2/ubuntu/manifests/latest \
  -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
  | python -mjson.tool

You are requesting a single platform manifest, so the registry will automatically dereference a multi-platform image manifest. Checking that digest from Hub, that shows up as the following:

$ regctl manifest get \
  ubuntu@sha256:9c152418e380c6e6dd7e19567bb6762b67e22b1d0612e4f5074bda6e6040c64a \
  --format '{{jsonPretty .}}'
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1463,
    "digest": "sha256:2b4cba85892afc2ad8ce258a8e3d9daa4a1626ba380677cee93ef2338da442ab"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 28565751,
      "digest": "sha256:7c3b88808835aa80f1ef7f03083c5ae781d0f44e644537cd72de4ce6c5e62e00"
    }
  ]
}

You can better see what's happening by pulling the manifest list by allowing an additional media type:

curl -k -v -s -X GET -u admin:root \
  https://docker.registry.url/v2/ubuntu/manifests/latest \
  -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
  -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
  | python -mjson.tool

An example of that looks like:

$ regctl manifest get ubuntu --format 'body' | jq .
{
  "manifests": [
    {
      "digest": "sha256:965fbcae990b0467ed5657caceaec165018ef44a4d2d46c7cdea80a9dff0d1ea",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 529
    },
    {
      "digest": "sha256:ea8f467d512068a1e52494d5b2d959a9307a35682633d0b5d481e79c914c627f",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 529
    },
    {
      "digest": "sha256:e77aa65a8a2bccbc47b96b4256995dd7ff447024ed5319527040f7cc465f6511",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "size": 529
    },
    {
      "digest": "sha256:48d0b40359f8789f1c5d6065bf0d2a00c8d0d4913ac61670215b2debedd483e6",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },
      "size": 529
    },
    {
      "digest": "sha256:f31546bc71659c643837d57f09a161f04e866b59da4f418e064082a756c4c23a",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "riscv64",
        "os": "linux"
      },
      "size": 529
    },
    {
      "digest": "sha256:4e3c33035b808801fc84ba48d5a779cd572167956f5123d846d7e1cbafe9e785",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 529
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

Since your registry is in a corrupt state, you're likely best off deleting the images and starting over, without the GC process. But you can copy individual images back to the server by digest:

$ regctl image copy \
  ubuntu@sha256:9c152418e380c6e6dd7e19567bb6762b67e22b1d0612e4f5074bda6e6040c64a \
  docker.registry.url/ubuntu@sha256:9c152418e380c6e6dd7e19567bb6762b67e22b1d0612e4f5074bda6e6040c64a

I even have the flag --force-recursive for this because I've seen it happen too often:

$ regctl image copy --help
...

Flags:
      --digest-tags        Include digest tags ("sha256-<digest>.*") when copying manifests
      --force-recursive    Force recursive copy of image, repairs missing nested blobs and manifests
  -h, --help               help for copy
      --include-external   Include external layers
      --referrers          Include referrers
...

$ regctl image copy --force-recursive \
  ubuntu:latest docker.registry.url/ubuntu:latest

The above is shown with my own regclient/regctl tool, but there's also crane, scopeo, and I'm sure several other tools including whichever you used to first copy the image yourself. I'd avoid using curl to pull the images yourself for more than a demo since you'll eventually run into scenarios other media types or token authentication that make the scripting around the curl command very complicated (I started down that path before deciding it was easier to write regclient).