How to skip TLS cert check for crictl (containerd CR) while pulling the images from private repository

Question

I have installed k8s 1.24 version and containerd (containerd://1.5.9) is the CR for my setup (ubuntu 20.04).

I have also installed docker on my VM and have added my private repository under /etc/docker/daemon.json with the following changes:

{   "insecure-registries" : ["myPvtRepo.com:5028"] }

When I am running docker pull myPvtRepo:123/image after login to my pvt repo by using docker login myPvtRepo:123 command, I am able to pull the images while running the same command with crictl pull myPvtRepo:123/image, I am facing:

E0819 06:49:01.200489 162610 remote_image.go:218] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image "myPvtRepo.com:5028/centos:latest": failed to resolve reference "myPvtRepo.com:5028/centos:latest": failed to do request: Head https://myPvtRepo.com::5028/v2/centos/manifests/latest: x509: certificate signed by unknown authority" image="myPvtRepo.com::5028/centos:latest" FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "myPvtRepo.com::5028/centos:latest": failed to resolve reference "myPvtRepo.com:5028/centos:latest": failed to do request: Head https://myPvtRepo.com::5028/v2/centos/manifests/latest: x509: certificate signed by unknown authority

FYI, I have modified /etc/containerd/config.toml with below content.

version = 2

[plugin."io.containerd.grpc.v1.cri".registry.configs."myPvtRepo.com:5028".tls]
    insecure_skip_verify = true

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
     endpoint = ["https://myPvtRepo.com:5028", "https://myPvtRepo.com:5038", "https://myPvtRepo.com:5037",
 "https://myPvtRepo.com:5039"]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."IP:5000"]
     endpoint = ["http://IP:5000"]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."IP:5000"]
     endpoint = ["http://IP:5000"]

I have also modified containerd's endpoint to point to containerd's sock.

Can you please help me out to understand and fix that even after setting insecure_skip_verify = true for my pvt repository and restarting the containerd service why I am getting this issue.

Answer 1

You will need to specify the hosts.toml file for the private registry and add skip-verify = true.

ref: https://github.com/containerd/containerd/blob/main/docs/hosts.md

Steps:

1. create folders: mkdir -p /etc/containerd/certs.d/<your registry>
2. add these config in /etc/containerd/config.toml:

   [plugins."io.containerd.grpc.v1.cri".registry]
     config_path = "/etc/containerd/certs.d"
   

3. create and edit hosts.toml under the just created folder

   server = "https://<your registry>"
   
   
   [host."https://<your registry>"]
     capabilities = ["pull", "resolve"]
     skip_verify = true
   

Answer 2

I got a solution:

cd /usr/local/share/ca-certificates/
curl -L --remote-name http://your-artifacts.com/xyz-bundle.crt
/usr/sbin/update-ca-certificates

This one work for me.

Also make sure to update your endpoints under /etc/crictl.yaml

runtime-endpoint: unix:///run/containerd/containerd.sock 
image-endpoint: "" 
timeout: 0 
debug: false 
pull-image-on-create: false 
disable-pull-on-run: false