Since today I haven't been able to push new signed images to a Docker Hub private repository due to image signing failing. I have Docker Content Trust enabled. I don't know of any significant changes in my environment that could affect this, except routinely installing the latest security updates to Ubuntu a couple days ago. But signed image pushing did work after those upgrades.
My question is, how to go about debugging signing related problems? There seem to be not much available by googling or duckduckgoing.
I tried running the notary CLI but it didn't seem to provide much help to me. The various options of different commands are not very well documented.
Environment:
- OS: Ubuntu 18.04.1 LTS
- Docker version 18.06.1-ce, build e68fc7a
relevant environment variables:
DOCKER_CONTENT_TRUST=1 DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE=[undisclosed] DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE=[undisclosed]
Failing command:
docker push xyz/abc:def
Sample output (obfuscated from the unrelevant parts):
user@machine:~$ source .docker-signing-credentials
user@machine:~$ export DOCKER_CONTENT_TRUST=1
user@machine:~$ docker push myorg/myproject:myimage_v1.38.0
The push refers to repository [docker.io/myorg/myproject]
c72506834af4: Layer already exists
043ae531d76e: Layer already exists
... 8< ... snip ... 8< ...
af840f32f0a2: Layer already exists
8decd5535924: Layer already exists
myimage_v1.38.0: digest: sha256:baa3e1148e0100df8cbb0aab46200be2bdf600d7802d7cddb3a23c12053af82d size: 8883
Signing and pushing trust metadata
failed to sign docker.io/myorg/myproject:myimage_v1.38.0: An error occurred during validation: rpc error: code = 14 desc = grpc: RPC failed fast due to transport failure
When I unset DOCKER_CONTENT_TRUST, there is no problem with pushing the images.
There is an issue open with this same exact description in:
https://github.com/docker/hub-feedback/issues/1646
it might be a good idea to join to this issue.