buildkit-cli-for-kubectl does not put images on all k8s nodes

Question

I have a question regarding the Kubernetes Nodes images being available to other nodes when using buildkit-cli-for-kubectl. I have a weird case where I build an image with no problem. The image actually works on the node that it is sent to by "kubectl build". But putting affinity to one of the other K8s Nodes does not work, I get "ImagePullBackOff". I guess I was assuming that nodes will get from each other any images they lack but that is not the case.

When running this command on SRV-008 it exports the layers to SRV-009 but I have 3 modes in my cluster. How can they have built this for only transferring to one node? I find it weird because if it can transfer to one of the nodes in the K8s cluster then why not the others?

root@SRV-008:/mnt/kubernetes/ubuntu1604_jdk8# kubectl build -f Dockerfile . \
--output type=image,name=docker.io/library/ubuntu1604_jdk8:1.0.0

Is there something I am missing? I have looked in all the documentation about buildctl but nothing about how to get the built image onto all K8s nodes.

Any help would be appreciated.

Answer 1

So I figured out how to do this. below is all I did, of course change to your needs.

Blog I found where the answer was hiding.

https://blogs.vmware.com/opensource/2020/11/17/buildkit-cli-for-kubectl/

If you have a small multi-node cluster where you want to build images, 
and then deploy them in the same cluster, you can scale the number of 
builders to match the number of nodes. Each of the images will be 
replicated onto every node which has a builder.

Where the project lives

https://github.com/vmware-archive/buildkit-cli-for-kubectl

I am working with Linux servers. The key here is the replicas count!

    wget https://github.com/vmware-archive/buildkit-cli-for-kubectl/releases/download/v0.1.6/linux-v0.1.6.tgz
    cat linux-v0.1.6.tgz | tar -C /usr/local/bin -zxvf -
    kubectl build --help (Should be avaialble.)
    kubectl buildkit create --config /tmp/buildkitd.toml --replicas 3

cd to the folder where your Dockefile is located.

kubectl build -f Dockerfile . --output type=image,name=ubuntu1604_jdk8:1.0.0

In each of the K8s Nodes yo will see the image you just built.

ctr --namespace=k8s.io images ls | grep ubuntu1604

buildkitd.toml file contents. Make any other changes you need here.

#kubectl buildkit create --config ./buildkitd.toml
debug = false
[worker.containerd]
  namespace = "k8s.io"

Command to delete the buildkit pods after you build your image.

kubectl delete deployment.apps/buildkit