Mesos cannot deploy container from private Docker registry

Question

I have a private Docker registry that is accessible at https://docker.somedomain.com (over standard port 443 not 5000). My infrastructure includes a set up of Mesosphere, which have docker containerizer enabled. I'm am trying to deploy a specific container to a Mesos slave via Marathon; however, this always fails with Mesos failing the task almost immediately with no data in stderr and stdout of that sandbox.

I tried deploying from an image from the standard Docker Registry and it appears to work fine. I'm having trouble figuring out what is wrong. My private Docker registry does not require password authentication (turned off for debugging this), AND if I shell into the Meso's slave instance, and sudo su as root, I can run a 'docker pull docker.somedomain.com/services/myapp' successfully every time.

Here is my Marathon post data for starting the task:

{
  "id": "myapp",
  "cpus": 0.5,
  "mem": 64.0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "docker.somedomain.com/services/myapp:2",
      "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 7000, "hostPort": 0, "servicePort": 0, "protocol": "tcp" }
      ]
    },
    "volumes": [
      {
        "containerPath": "application.yml",
        "hostPath": "/var/myapp/application.yml",
        "mode": "RO"
      }
    ]
  },
  "healthChecks": [
    {
      "protocol": "HTTP",
      "portIndex": 0,
      "path": "/",
      "gracePeriodSeconds": 5,
      "intervalSeconds": 20,
      "maxConsecutiveFailures": 3
    }
  ]
}   

I've been stuck on this for almost a day now, everything I've tried seems to be yielding the same result. Any insights on this would be much appreciated.

My versions: Mesos: 0.22.1 Marathon: 0.8.2 Docker: 1.6.2

Answer 1

If it is a problem between Marathon and the registry, the answer should be in the http logs of your registry. If Marathon connects, there will be an entry. And the Mesos master log should contain a clue as well.

It doesn't really sound like a problem between Marathon and Registry though. Are you sure you have 'docker,mesos' in /etc/mesos-slave/containerizers?

Answer 2

So this turns out to be an issue with volumes

"volumes": [
      {
        "containerPath": "/application.yml",
        "hostPath": "/var/myapp/application.yml",
        "mode": "RO"
      }
    ]

Using the root path of the container of the root path may be legal in docker, but Mesos appears not to handle this behavior. Modifying the containerPath to a non-root path resolves this, i.e

"volumes": [
      {
        "containerPath": "/var",
        "hostPath": "/var/myapp",
        "mode": "RW"
      }
    ]

Answer 3

Did you --despite having no authentification-- try to follow Using a Private Docker Repository?

To supply credentials to pull from a private repository, add a .dockercfg to the uris field of your app. The $HOME environment variable will then be set to the same value as $MESOS_SANDBOX so Docker can automatically pick up the config file.