Docker Trust Initialization

Question

When the initial trust on docker content trust with notary on tuf is initialized I understand how TUF, Notary and Content Trust works.

But what is not clear to me is, how the initial trust is setup.

How do I know, that the first pull is not a compromised one and the initial root.json is trustworthy?

So for example if I do docker pull with content-trust enabled, I will only get signed images. But how do I verify, that this image is signed by the right person?

Answer 1

Notary creator and maintainer here. Justin has already given a good account but I'll speak to trust initialization in TUF and Notary more broadly.

Unless you communicate the root of trust through some out of band method, there will always be a point of download that you implicitly trust to deliver the root of trust. Some general case examples: we do this when we download an OS (i.e. any Linux distro), or grab somebody's GPG public key from a public key directory. Assuming the resources are delivered over a TLS connection and we believe that the publisher has secured their server, we trust we're receiving legitimate data, and use this to bootstrap trust on all future interactions. This is called Trust On First Use, or TOFU.

The claim here is that people do keep their servers secure, and that it's difficult to perform a Man-in-the-middle (MiTM) attack, especially against a TLS secured connection. Therefore we can trust this initial download.

Notary has a few ways one can initialize trust. The first is this TOFU mechanism. TUF has a defined update flow that ensures trust over all subsequent content after the initial download. Notary implements this update flow and ensures the publisher is consistent after the initial download.

If you want to additionally ensure the publisher is a specific entity, Notary provides three different ways to bootstrap that trust. They are:

1. Manually place the root.json, acquired out of band, in the correct location in the notary cache.
2. Configure trust pinning to trust a specific root key ID for a Notary Globally Unique Name (GUN).
3. Configure trust pinning to trust a CA for a specific Notary GUN or GUN prefix.

More information on trust pinning can be found in our docs. Note all 3 options require an out of band communication in which you acquire either a root.json, the ID of the root key, or the CA certificate that was used to issue the root key.

Implementing trust pinning under the docker trust command is in our TODO list, it's not there yet. However you can still use option 1 with docker trust. The cache is located at ~/.docker/trust/tuf/<GUN>/metadata/

Additional context on option 3: Notary implements a feature that allows one to configure CAs for GUNs or GUN prefixes. The requirement in this instance is that the public root key is included in the root.json as an x509 certificate that chains to the configured CA. While CAs can be a controversial topic, nobody is forced to use this feature and in most attacker models it's strictly better than TOFU. Additionally TUF explicitly does not address how trust is bootstrapped.

Answer 2

tl;dr

You can execute the following to pin the public root key for debian:

sudo su -
mkdir -p /root/.docker/trust/tuf/docker.io/library/debian/metadata
chown -R root:root /root/.docker
chmod -R 0700 /root/.docker
echo '{"signed":{"_type":"Root","consistent_snapshot":false,"expires":"2025-08-07T20:55:22.677722315-07:00","keys":{"5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsslGF2xHOYztrocb2OsRF2zth16v170QiLAyKdce1nQgOJ34FOk679ClPL9/RNnJukf2JfQXSlVV/qcsvxV2dQ=="}},"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84":{"keytype":"ecdsa-x509","keyval":{"private":null,"public":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lRWExkUFFHTGJaOE84UXFlTzVuZlBRekFLQmdncWhrak9QUVFEQWpBak1TRXcKSHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrWldKcFlXNHdIaGNOTVRVd09ERXhNRE0xTlRJeQpXaGNOTWpVd09EQTRNRE0xTlRJeVdqQWpNU0V3SHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrClpXSnBZVzR3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVE1ZGkxcmxPQjBMQmRNS2N0VFQxYmwKUGd6aXYxOUJDdW9tNEFNL3BUdURtdjBnS0E5S1ptNUVjLy9VQmhSODVCYmR0cTk0cXhQM3IwUjhRc3FQV1Y4SQpvelV3TXpBT0JnTlZIUThCQWY4RUJBTUNBS0F3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdNd0RBWURWUjBUCkFRSC9CQUl3QURBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBOUFOZ3dPN2tBdUVIK3U2N25XNlFLWmlMdWd5UVcKaEQ3Vys5WjIza01mTndJaEFJa3RTaW1TdFdRQkFoOG9WOXhjaWNVWWVUN0pyUG82a0RqeHU1YitGZ3MxCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}},"728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAENtpBkDJ2oYaAAVdOkP0A6J0XwUkYGuFRk+q8N4WCPu2VnNIuBJkatPCWdEtHfQ9nNYLeanWgG62/UmJnx3E2Yg=="}},"d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEwfs26T/cpjvNTXVJpK7Wv8oDOnNKL78AT3Y1QD356OIAggwPupX2LQjZU6CVzCjm+pkJIO4clu9Q2n540gKuzQ=="}}},"roles":{"root":{"keyids":["575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84"],"threshold":1},"snapshot":{"keyids":["d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69"],"threshold":1},"targets":{"keyids":["5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a"],"threshold":1},"timestamp":{"keyids":["728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70"],"threshold":1}},"version":1},"signatures":[{"keyid":"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9f
fb1ab5cc4291e07bf84","method":"ecdsa","sig":"3WbX1VXN9E8LRmSG+E4SQlBUNqBNchhwAStWnRWLLyAOoFNBq5xmIgSO3UYYuKyJvL7kbMoONRbn5Vk2p2Wqrg=="}]}' > /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json

export DOCKER_CONTENT_TRUST=1
docker pull debian:stable-slim

Long Answer

⚠ Disclaimer: I am not a docker developer. As of 2021, it appears that DCT is broken out-of-the-box and is far from being useful. My answer here is a best-guess, but I have not confirmed with the docker team if this is the "correct" way to pre-load and pin a given publisher's root key into the DCT keyring.

Be advised, proceed with caution, and your comments are very welcome.

It doesn't appear to be documented anywhere, but per this question it's clear that docker puts its DCT metadata (including root public keys) in the following location:

$HOME/.docker/trust/tuf/docker.io/library

Inside this library dir exists one dir per publisher. For the purposes of this answer, I'll use debian as our example publisher.

You can see the list of the debian docker images published to Docker Hub here:

• https://hub.docker.com/_/debian/

Solution

Let's say we want to download the stable-slim image from the debian publisher on Docker Hub. In this example, we'll also use a fresh install of Debian 10 as the docker host.

##
# first, install docker
## 
root@disp2716:~# apt-get install docker.io
...
root@disp2716:~#

##
# confirm that there is no docker config dir yet
##

root@disp2716:~# ls -lah /root/.docker
ls: cannot access '/root/.docker': No such file or directory
root@disp2716:~# 

##
# add the debian publisher's root DCT key
##

root@disp2716:~# mkdir -p /root/.docker/trust/tuf/docker.io/library/debian/metadata
root@disp2716:~# chown -R root:root /root/.docker
root@disp2716:~# chmod -R 0700 /root/.docker
root@disp2716:~# echo '{"signed":{"_type":"Root","consistent_snapshot":false,"expires":"2025-08-07T20:55:22.677722315-07:00","keys":{"5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsslGF2xHOYztrocb2OsRF2zth16v170QiLAyKdce1nQgOJ34FOk679ClPL9/RNnJukf2JfQXSlVV/qcsvxV2dQ=="}},"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84":{"keytype":"ecdsa-x509","keyval":{"private":null,"public":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lRWExkUFFHTGJaOE84UXFlTzVuZlBRekFLQmdncWhrak9QUVFEQWpBak1TRXcKSHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrWldKcFlXNHdIaGNOTVRVd09ERXhNRE0xTlRJeQpXaGNOTWpVd09EQTRNRE0xTlRJeVdqQWpNU0V3SHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrClpXSnBZVzR3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVE1ZGkxcmxPQjBMQmRNS2N0VFQxYmwKUGd6aXYxOUJDdW9tNEFNL3BUdURtdjBnS0E5S1ptNUVjLy9VQmhSODVCYmR0cTk0cXhQM3IwUjhRc3FQV1Y4SQpvelV3TXpBT0JnTlZIUThCQWY4RUJBTUNBS0F3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdNd0RBWURWUjBUCkFRSC9CQUl3QURBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBOUFOZ3dPN2tBdUVIK3U2N25XNlFLWmlMdWd5UVcKaEQ3Vys5WjIza01mTndJaEFJa3RTaW1TdFdRQkFoOG9WOXhjaWNVWWVUN0pyUG82a0RqeHU1YitGZ3MxCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}},"728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAENtpBkDJ2oYaAAVdOkP0A6J0XwUkYGuFRk+q8N4WCPu2VnNIuBJkatPCWdEtHfQ9nNYLeanWgG62/UmJnx3E2Yg=="}},"d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEwfs26T/cpjvNTXVJpK7Wv8oDOnNKL78AT3Y1QD356OIAggwPupX2LQjZU6CVzCjm+pkJIO4clu9Q2n540gKuzQ=="}}},"roles":{"root":{"keyids":["575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84"],"threshold":1},"snapshot":{"keyids":["d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69"],"threshold":1},"targets":{"keyids":["5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a"],"threshold":1},"timestamp":{"keyids":["728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70"],"threshold":1}},"version":1},"signatures":[{"keyid":"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84","method":"ecdsa","sig":"3WbX1VXN9E8LRmSG+E4SQlBUNqBNchhwAStWnRWLLyAOoFNBq5xmIgSO3UYYuKyJvL7kbMoONRbn5Vk2p2Wqrg=="}]}' > /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~# 
root@disp2716:~# chown root:root /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~# chmod 0600 /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~# 

##
# pull the docker image with DCT verification
##

root@disp2716:~# export DOCKER_CONTENT_TRUST=1
root@disp2716:~# docker pull debian:stable-slim
Pull (1 of 1): debian:stable-slim@sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632
sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632: Pulling from library/debian
6e640006d1cd: Pull complete 
Digest: sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632
Status: Downloaded newer image for debian@sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632
Tagging debian@sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632 as debian:stable-slim
root@disp2716:~# 

Proof

While there's no way to tell docker to fail on TOFU, we can confirm that the above key pinning works by making the public key something else

##
# first, move the docker config dir out of the way
## 

mv /root/.docker /root/.docker.bak

##
# add the debian publisher's root DCT key (note I just overwrote the first 8
# characters of the actual key with "INVALID/")
##

root@disp2716:~# mkdir -p /root/.docker/trust/tuf/docker.io/library/debian/metadata
root@disp2716:~# chown -R root:root /root/.docker
root@disp2716:~# chmod -R 0700 /root/.docker
root@disp2716:~# echo '{"signed":{"_type":"Root","consistent_snapshot":false,"expires":"2025-08-07T20:55:22.677722315-07:00","keys":{"5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a":{"keytype":"ecdsa","keyval":{"private":null,"public":"INVALID/KoZIzj0CAQYIKoZIzj0DAQcDQgAEsslGF2xHOYztrocb2OsRF2zth16v170QiLAyKdce1nQgOJ34FOk679ClPL9/RNnJukf2JfQXSlVV/qcsvxV2dQ=="}},"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84":{"keytype":"ecdsa-x509","keyval":{"private":null,"public":"INVALID/RUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lRWExkUFFHTGJaOE84UXFlTzVuZlBRekFLQmdncWhrak9QUVFEQWpBak1TRXcKSHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrWldKcFlXNHdIaGNOTVRVd09ERXhNRE0xTlRJeQpXaGNOTWpVd09EQTRNRE0xTlRJeVdqQWpNU0V3SHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrClpXSnBZVzR3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVE1ZGkxcmxPQjBMQmRNS2N0VFQxYmwKUGd6aXYxOUJDdW9tNEFNL3BUdURtdjBnS0E5S1ptNUVjLy9VQmhSODVCYmR0cTk0cXhQM3IwUjhRc3FQV1Y4SQpvelV3TXpBT0JnTlZIUThCQWY4RUJBTUNBS0F3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdNd0RBWURWUjBUCkFRSC9CQUl3QURBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBOUFOZ3dPN2tBdUVIK3U2N25XNlFLWmlMdWd5UVcKaEQ3Vys5WjIza01mTndJaEFJa3RTaW1TdFdRQkFoOG9WOXhjaWNVWWVUN0pyUG82a0RqeHU1YitGZ3MxCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}},"728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70":{"keytype":"ecdsa","keyval":{"private":null,"public":"INVALID/KoZIzj0CAQYIKoZIzj0DAQcDQgAENtpBkDJ2oYaAAVdOkP0A6J0XwUkYGuFRk+q8N4WCPu2VnNIuBJkatPCWdEtHfQ9nNYLeanWgG62/UmJnx3E2Yg=="}},"d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69":{"keytype":"ecdsa","keyval":{"private":null,"public":"INVALID/KoZIzj0CAQYIKoZIzj0DAQcDQgAEwfs26T/cpjvNTXVJpK7Wv8oDOnNKL78AT3Y1QD356OIAggwPupX2LQjZU6CVzCjm+pkJIO4clu9Q2n540gKuzQ=="}}},"roles":{"root":{"keyids":["575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84"],"threshold":1},"snapshot":{"keyids":["d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69"],"threshold":1},"targets":{"keyids":["5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a"],"threshold":1},"timestamp":{"keyids":["728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70"],"threshold":1}},"version":1},"signatures":[{"keyid":"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84","method":"ecdsa","sig":"3WbX1VXN9E8LRmSG+E4SQlBUNqBNchhwAStWnRWLLyAOoFNBq5xmIgSO3UYYuKyJvL7kbMoONRbn5Vk2p2Wqrg=="}]}' > /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~# 
root@disp2716:~# chown root:root /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~# chmod 0600 /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~# 

##
# pull the docker image with DCT verification
##

root@disp2716:~# export DOCKER_CONTENT_TRUST=1
root@disp2716:~# docker pull debian:stable-slim
could not validate the path to a trusted root: unable to retrieve valid leaf certificates
root@disp2716:~# 
root@disp2716:~# echo $?
1
root@disp2716:~# 

Note that docker exits 1 with an error refusing to pull the debian:stable-slim docker image from Docker Hub because it cannot trust its signature

Answer 3

You can pin the keys by some out of band means, or do something like ssh which shows you a key to check on first use. These methods are not predefined, but you have the flexibility to build them yourself depending how you are using Notary. For LinuxKit we are planning to have an option to put the key hashes in the config file you use for building, that lists which images to pull. Or you could publish the root key id elsewhere.