docker multi-stage build Go image - x509: certificate signed by unknown authority

Question

I try to build go images in private corp network use docker-multi-stage-build:

FROM golang:latest as builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN  GO111MODULE="on" CGO_ENABLED=0 GOOS=linux go build -o main ${MAIN_PATH}

FROM alpine:latest
LABEL maintainer="Kozmo"
RUN apk add --no-cache bash
WORKDIR /app
COPY --from=builder /app/main .
EXPOSE 8080
CMD ["./main"]

and get x509: certificate signed by unknown authority error

Step 1/13 : FROM golang:latest as builder
 ---> 2421885b04da
Step 2/13 : WORKDIR /app
 ---> Using cache
 ---> 6555644dbd16
Step 3/13 : COPY go.mod go.sum ./
 ---> 55d45a30f492
Step 4/13 : RUN go mod download
 ---> Running in 88c21c6b4fab
go: github.com/dgrijalva/jwt-go/[email protected]: Get "https://proxy.golang.org/github.com/dgrijalva/jwt-go/v4/@v/v4.0.0-preview1.mod": x509: certificate signed by unknown authority
The command '/bin/sh -c go mod download' returned a non-zero code: 1
make: *** [docker] Error 1

I tried to find an answer in

X509: Certificate Signed by Unknown Authority (Running a Go App Inside a Docker Container)

and

docker build: cannot get the github public repository, x509: certificate signed by unknown authority

and

x509 certificate signed by unknown authority - go-pingdom

, but result is the same.

---

❗️If add -insecure flag

...
RUN go env -w GOPROXY=direct GOFLAGS="-insecure"
COPY go.mod go.sum ./
...

to Dockerfile unrecognized import path error wrap previous x509 error and an unreachable package change to golang.org/x/crypto

go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/crypto": https fetch: Get "https://golang.org/x/crypto?go-get=1": x509: certificate signed by unknown authority

What is the problem❓

(I understand that problem is in the certificates and authentication when git get dependencies, but I try to make process of building images more common)

Answer 1

git uses curl to access the https servers so you need to import the certificate into the CA store of the system.

The workaround is to define the environment variable GIT_SSL_NO_VERIFY=1 on your Agent environment variables, but it doesn't work when using go get or go mod download .

To import the certificate on your system CA store the procedure depends on your OS you have to use openssl.

For example

FROM golang:latest as builder

RUN apt-get update && apt-get install -y ca-certificates openssl

ARG cert_location=/usr/local/share/ca-certificates

# Get certificate from "github.com"
RUN openssl s_client -showcerts -connect github.com:443 </dev/null 2>/dev/null|openssl x509 -outform PEM > ${cert_location}/github.crt
# Get certificate from "proxy.golang.org"
RUN openssl s_client -showcerts -connect proxy.golang.org:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >  ${cert_location}/proxy.golang.crt
# Update certificates
RUN update-ca-certificates

WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN  GO111MODULE="on" CGO_ENABLED=0 GOOS=linux go build -o main ${MAIN_PATH}

FROM alpine:latest
LABEL maintainer="Kozmo"
RUN apk add --no-cache bash
WORKDIR /app
COPY --from=builder /app/main .
EXPOSE 8080
CMD ["./main"]

docker image build output

...

Step 5/19 : RUN openssl s_client -showcerts -connect github.com:443 </dev/null 2>/dev/null|openssl x509 -outform PEM > ${cert_location}/github.crt
 ---> Running in bb797e26d4b4
Removing intermediate container bb797e26d4b4
 ---> 6c68ddafd884
Step 6/19 : RUN openssl s_client -showcerts -connect proxy.golang.org:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >  ${cert_location}/proxy.golang.crt
 ---> Running in 61f59939d75e
Removing intermediate container 61f59939d75e
 ---> 72d2b03b11e6
Step 7/19 : RUN update-ca-certificates
 ---> Running in 6cf9aa248776
Updating certificates in /etc/ssl/certs...
2 added, 0 removed; done.  'certificates updated'

...

Step 8/18 : COPY go.mod go.sum ./
 ---> 436263b76050
Step 9/18 : RUN go mod download  'works fine'
 ---> Running in 2387c78147db
Removing intermediate container 2387c78147db
 ---> a37c05c2b531
Step 10/18 : COPY . .
 ---> 01b49c388f59

...

Answer 2

Coping self certificates (.crt) helped

1️⃣ add .crt to required dir

.
└── backend
    ├── Dockerfile
    ├── Makefile
    ├── cmd
    │   └── main.go
    ├── etc
    │   ├── ssl
    │   │   └── github.crt #❗️a copy of the self certificate 

2️⃣ COPY certificates to 'builder'-container

FROM golang:latest as builder
COPY  etc/ssl/ /etc/ssl/certs/ #❗️add certificates to the container 
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download

---

Answer 3

I'll suggest a couple of things:

• Build your code within the same OS distribution as the final code image, so that you are sure that your code will run in that specific distribution. Also some distributions require certs to be in different folders, so be aware of that.
• Using alpine for the first image will substantially decrease your build time. You can see here latest size is ~260M, but alpine is ~100M.
• Better will be to use an specific version of alpine, so that you can be sure that your code runs in that version (I leave this at your discretion)
• Something very powerful of Golang is that you can run it in an empty docker image called scratch, this means, your final docker images does not contain more than your own executable.
• If you need your own certificates you must have them in your code and copy them before executing update-ca-certificates so that they get included in the final file

Here's an example of the dockerfile with what I explained above

FROM golang:alpine as builder
WORKDIR /app

# This will download all certificates (ca-certificates) and builds it in a
# single file under /etc/ssl/certs/ca-certificates.crt (update-ca-certificates)
# I also add git so that we can download with `go mod download` and
# tzdata to configure timezone in final image
RUN apk --update add --no-cache ca-certificates openssl git tzdata && \
update-ca-certificates

COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN  GO111MODULE="on" CGO_ENABLED=0 GOOS=linux go build -o main ${MAIN_PATH}

# Golang can run in a scratch image, so that, the only thing that your docker 
# image contains is your executable
FROM scratch
LABEL maintainer="Kozmo"
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo

# This line will copy all certificates to final image
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
WORKDIR /app
COPY --from=builder /app/main .
EXPOSE 8080
CMD ["./main"]

If own certificates replace first docker stage with:

FROM golang:alpine as builder
WORKDIR /app

RUN apk --update add --no-cache ca-certificates openssl git tzdata

COPY your/cert/path /usr/local/share/ca-certificates/your-cert-name

RUN update-ca-certificates

COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN  GO111MODULE="on" CGO_ENABLED=0 GOOS=linux go build -o main ${MAIN_PATH}

Because you use own certificates your final Dockerfile will look like this:

FROM golang:alpine as builder
WORKDIR /app

RUN apk --update add --no-cache ca-certificates openssl git tzdata

COPY your/cert/path /usr/local/share/ca-certificates/your-cert-name

RUN update-ca-certificates

COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN  GO111MODULE="on" CGO_ENABLED=0 GOOS=linux go build -o main ${MAIN_PATH}

FROM scratch
LABEL maintainer="Kozmo"
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo

# This line will copy all certificates to final image
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
WORKDIR /app
COPY --from=builder /app/main .
EXPOSE 8080
CMD ["./main"]

Feel free to ask me if you have any doubt :)

Answer 4

Docker-Private-Registry

Creating your own Private Docker Registry (Ubuntu 18.04 Linux) with self signed TLS Certificate

Pre-Req : Ubuntu VM with Docker-CE Edition installed on it and required ports open to the outside world (or as required)

Add your Ubuntu VM IP address in subjectAltName in the openssl.cnf before generating certficates

sudo vi /etc/ssl/openssl.cnf

Add the following with your VM specific IP address under the section [ v3_ca ]

[ v3_ca ]
subjectAltName=IP:IP_ADDRESS_OF_YOUR_VM

Create a local folder which will hold the certificates and that can be referenced by the Docker Registry server

mkdir -p /certificates

cd certificates

openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout domain.key \
  -x509 -days 365 -out domain.crt
  
#Enter all required fields (it could be anything) but please enter your Server IP address when it prompts for -> Common Name (e.g. server FQDN or YOUR name)

Common Name (e.g. server FQDN or YOUR name) []: IP_ADDRESS_OF_YOUR_VM

# Check if the certificates are created in the current directory (certificates)

ls

Launch Docker registry using version 2 and referencing the certificates folder for TLS

sudo docker run -d -p 5000:5000 --restart=always --name registry \
  -v /certificates:/certificates \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certificates/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certificates/domain.key \
  registry:2
  
docker ps
docker logs CONTAINER-ID

#Check & proceed further if there are no errors in the registry container log

To verify our Docker registry, let us pull a small hello-world docker image from Docker-Hub registry, tag it appropriately and try to push it to our local Registry

docker pull hello-world
docker tag hello-world IP_ADDRESS_OF_YOUR_VM:5000/hello-world

docker push IP_ADDRESS_OF_YOUR_VM:5000/hello-world

Certificate Error? Docker Server does not trust the self-signed certificate and our certificates need to be manually added to Docker Engine

sudo mkdir -p /etc/docker/certs.d/IP_ADDRESS_OF_YOUR_VM:5000

#Please make sure to copy domain.crt as ca.crt (or rename later on) 
sudo cp /certificates/domain.crt /etc/docker/certs.d/IP_ADDRESS_OF_YOUR_VM:5000/ca.crt

sudo ls /etc/docker/certs.d/IP_ADDRESS_OF_YOUR_VM:5000/

#reload docker daemon to use the ca.crt certificate
sudo service docker reload

Try pushing to the local registry server now!!

Answer 5

From your error message

Get "https://proxy.golang.org/github.com/dgrijalva/jwt-go/v4/@v/v4.0.0-preview1.mod": x509: certificate signed by unknown authority

It looks like the CA root of proxy.golang.org is not part of the trusted root CAs in your private corp docker environment.

I'd try to install it with :

1 - Get the certificate from proxy.golang.org :

echo -n | openssl s_client -connect proxy.golang.org:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ./golang.cer

If you open golang.cer you should see the certificates chain

2 - install it in your trusted root CAs :

certutil.exe -addstore root golang.cer

...or on Mac :

2a - Double click the certificate file (with ".cer" extension)

2b - Choose "System" from the keychain option. Then press "OK"

2c - When the following window pops-up, click the "Always Trust" button.