How to make GitLab Runner in Docker see a custom CA Root certificate

Question

I have installed and configured:

1. an on-premises GitLab Omnibus on ServerA running on HTTPS
2. an on-premises GitLab-Runner installed as Docker Service in ServerB

ServerA certificate is generated by a custom CA Root

The Configuration

I've have put the CA Root Certificate on ServerB:

/srv/gitlab-runner/config/certs/ca.crt

Installed the Runner on ServerB as described in Run GitLab Runner in a container - Docker image installation and configuration:

docker run -d --name gitlab-runner --restart always \
           -v /srv/gitlab-runner/config:/etc/gitlab-runner \
           -v /var/run/docker.sock:/var/run/docker.sock \
           gitlab/gitlab-runner:latest

Registered the Runner as described in Registering Runners - One-line registration command:

docker run --rm -t -i 
            -v /srv/gitlab-runner/config:/etc/gitlab-runner 
           --name gitlab-docker-runner gitlab/gitlab-runner register \
           --non-interactive \
           --executor "docker" \
           --docker-image alpine:latest \
           --url "https://MY_PRIVATE_REPO_URL_HERE/" \
           --registration-token "MY_PRIVATE_TOKEN_HERE" \
           --description "MyDockerServer-Runner" \
           --tag-list "TAG_1,TAG_2,TAG_3" \
           --run-untagged \
           --locked="false"

This command gave the following output:

Updating CA certificates...
Runtime platform arch=amd64 os=linux pid=5 revision=cf91d5e1 version=11.4.2
Running in system-mode.

Registering runner... succeeded runner=8UtcUXCY
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

I checked with

$ docker exec -it gitlab-runner bash 

and once in the container with

$ awk -v cmd='openssl x509 -noout -subject' '
/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

and the custom CA root is correctly there.

The Problem

When running Gitlab-Runner from GitLab-CI, the pipeline fails miserably telling me that:

$ git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git

Cloning into 'My-Project.wiki'...

fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@ServerA/foo/bar/My-Project.wiki.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

ERROR: Job failed: exit code 1

It does not recognize the Issuer (my custom CA Root), but according to The self-signed certificates or custom Certification Authorities, point n.1, it should out-of-the-box:

Default: GitLab Runner reads system certificate store and verifies the GitLab server against the CA’s stored in system.

I've then tried the solution from point n.3, editing

/srv/gitlab-runner/config/config.toml:

and adding:

[[runners]]
tls-ca-file = "/srv/gitlab-runner/config/certs/ca.crt"

But it still doesn't work.

How can I make Gitlab Runner read the CA Root certificate?

Answer 1

While I've still not got why it doesn't work out-of-the-box, I've found the Egg of Columbus:

Gitlab-Runner configuration:

[[runners]]
  name = "MyDockerServer-Runner"
  url = "https://MY_PRIVATE_REPO_URL_HERE/"
  token = "MY_TOKEN_HERE"
  executor = "docker"
  ...
  [runners.docker]
    image = "ubuntu:latest"

  # The trick is the following:
    volumes = ["/cache","/srv/gitlab-runner/config:/etc/gitlab-runner"]
    ...

Gitlab-ci.yml pipeline:

MyJob:
    image: ubuntu:latest

    script:
      - awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
      - git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git
      - wget -O foo.png https://ServerA/foo/bar/foo.png 

    before_script:
      - apt-get update -y >/dev/null
      - apt-get install -y apt-utils dialog >/dev/null
      - apt-get install -y git >/dev/null
      - apt-get install -y wget >/dev/null

    # The trick is the following:
      - cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
      - update-ca-certificates

That's it:

• Mount the volume once (per Docker executor)
• Update the CA certificates once (per job)

And everything will work as expected: git clone, wget https, etc...

A great workaround, until someone at GitLab will fix it or explain me where I'm wrong (be my guest!)

Answer 2

You have two options:

Ignore SSL verification

Put this at the top of your .gitlab-ci.yml:

variables:
  GIT_SSL_NO_VERIFY: "1"

Point GitLab-Runner to the proper certificate

As outlined in the official documentation, you can use the tls-*-file options to setup your certificate, e.g.:

[[runners]]
  ...
  tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"
  [runners.docker]
  ...

As the documentation states, "this file will be read every time when runner tries to access the GitLab server."

Other options include tls-cert-file to define the certificate to be used if needed.

Answer 3

From the output you provided i think that the certificate might be OK but you are lacking the CRL file : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

The CRL file is used to verify that even if the certificate is valid is hasn't been revoked by the CA owner. You shoudl then need to :

1) Generate a CRL file based on your CA:

openssl ca -gencrl -keyfile ca.key -cert ca.crt -out crl.pem

source: https://blog.didierstevens.com/2013/05/08/howto-make-your-own-cert-and-revocation-list-with-openssl/

2) Instruct the runner to use it :

[[runners]]
  ...
  tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"
  crl-file = "/etc/gitlab-runner/ssl/ca.crl"

3) Of course setting GIT_SSL_NO_VERIFY will work but you will be more sensitive to man-in-the-middle attacks

Answer 4

Not sure it's the best approach, but at least it worked for me. You can create a customized gitlab runner image and add your root CA inside:

├── Dockerfile
└── myca.crt

# Dockerfile
FROM gitlab/gitlab-runner:latest
COPY myca.crt /usr/local/share/ca-certificates
RUN update-ca-certificates

Build it:

docker build -t custom-gitlab-runner .

And rerun all your commands, just remember to use this new image name.

Off-topic, but related and might be useful

Dockerized gitlab-runner seem to also ignore entries in your /etc/hosts, so if you have launched Gitlab on a custom domain, e.g. https://gitlab.local.net, you need to pass the values from /etc/hosts when launching/registering gitlab runner:

docker run -d --name gitlab-runner --restart always \
       --add-host="gitlab.local.net:192.168.1.100" \
       ...

If you want to launch docker:dind (docker in docker service) container to build docker images, you also need to set these values inside /srv/gitlab-runner/config/config.toml:

[[runners]]
  url = "https://gitlab.local.net/"
  executor = "docker"
  pre_clone_script = "echo '192.168.1.100 gitlab.local.net registry.local.net' >> /etc/hosts"
  ...