Kubernetes with CRI-O and docker private registry: pods running only on master node

Question

I have a kubernetes cluster running on 3 VMs and I enabled master nodes for pods. I also have docker private registry running on another VM with valid SSL certificates installed. I am using CRI-O in my kubernetes cluster. This is what I am doing

1. VM with Jenkins server and kubctl configured so I can connect to the cluster remotely

2. Separate VM specifically for docker registry. Bought SSL cert from Godaddy and added in /etc/docker/certs.d

3. Created secret following this doc

4. Added certs in /etc/crio/certs.d on all nodes including master and two worker nodes. I am able to pull and push images from my jenkins VM and Kubernetes also seems working but only on Master node. The pod works perfectly fine in Master node but two worker nodes shows certificate error. They have ImagePullBackOff with below error

       Failed to pull image "imagehub.mydomain.com:443/iam-config-server:0.0.2": rpc error: code = Unknown desc = pinging container registry imagehub.mydomain.com:443: Get "https://imagehub.mydomain.com:443/v2/": x509: certificate signed by unknown authority.
   

Its a spring boot application and here is my deploy.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iamconfigserver-deploy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: iam-config-server
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: iam-config-server
    spec:
      containers:
      - name: iamconfigserver-pod
        image: imagehub.mydomain.com:443/iam-config-server:0.0.2
        ports:
        - containerPort: 8071
      imagePullSecrets:
      - name: regcred

svc.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iamconfigserver-deploy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: iam-config-server
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: iam-config-server
    spec:
      containers:
      - name: iamconfigserver-pod
        image: imagehub.mydomain.com:443/iam-config-server:0.0.2
        ports:
        - containerPort: 8071
      imagePullSecrets:
      - name: regcred

I could run this in master node and get the secert

kubectl get secret regcred --output=yaml


apiVersion: v1
data:
  .dockerconfigjson: ew..............Cgl9Cn0=
kind: Secret
metadata:
  creationTimestamp: "2022-03-24T06:20:44Z"
  name: regcred
  namespace: default
  resourceVersion: "471374"
  uid: 2e6ba870-asf3-33dd-8340-sdfsafsdfsd4
type: kubernetes.io/dockerconfigjson

I am not sure what I am missing here. My Kubernetes VMs are all running on a separate physical server including master node. But still confused why only pods are running successfully only on master node. Its a development environment and I do understand its not ideal to run pods on master node. A help would be really appreciated. I am not sure if the location of certs in CRI-O is accurate but still its working fine on master node.

Answer 1

This helped me out:

https://github.com/cri-o/cri-o/issues/1768 https://github.com/Nordix/xcluster/tree/master/ovl/private-reg

We need to define the local registry in CRIO conf file on the master and worker nodes.

Then after try to pull the image from defined registry from each node, it should work.

crictl -D pull registry-ip:port/imagename