I have an interesting situation where it seems I cannot use locally imported images (directly to ctr cache). The setup is a single node microk8s where I have moved and imported all the necessary images to spin up the environment. The main issue happens when I try to setup Cilium using helm (the helm chart is also transferred).
I can see the images are present within the ctr ls (checked that the namespace is k8s.io):
microk8s ctr --namespace k8s.io image ls | grep cilium
quay.io/cilium/cilium:v1.11.12 application/vnd.docker.distribution.manifest.list.v2+json sha256:67e6af41d4711bf3514ee1055b6b5b9a83b048c6224da78e7824550b3ec28940 152.9 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/cilium:v1.14.2@sha256:6263f3a3d5d63b267b538298dbeb5ae87da3efacf09a2c620446c873ba807d35** application/vnd.docker.distribution.manifest.list.v2+json sha256:6263f3a3d5d63b267b538298dbeb5ae87da3efacf09a2c620446c873ba807d35 174.1 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/cilium@sha256:67e6af41d4711bf3514ee1055b6b5b9a83b048c6224da78e7824550b3ec28940 application/vnd.docker.distribution.manifest.list.v2+json sha256:67e6af41d4711bf3514ee1055b6b5b9a83b048c6224da78e7824550b3ec28940 152.9 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/operator-generic:v1.11.12 application/vnd.docker.distribution.manifest.list.v2+json sha256:4a036edd5afce506dc5e0ed340902c017c166f16f0a9ab37d59031f26c88a9d0 16.0 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/operator-generic:v1.14.2@sha256:52f70250dea22e506959439a7c4ea31b10fe8375db62f5c27ab746e3a2af866d application/vnd.docker.distribution.manifest.list.v2+json sha256:52f70250dea22e506959439a7c4ea31b10fe8375db62f5c27ab746e3a2af866d 22.5 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
quay.io/cilium/operator-generic@sha256:4a036edd5afce506dc5e0ed340902c017c166f16f0a9ab37d59031f26c88a9d0 application/vnd.docker.distribution.manifest.list.v2+json sha256:4a036edd5afce506dc5e0ed340902c017c166f16f0a9ab37d59031f26c88a9d0 16.0 MiB linux/amd64,linux/arm64 io.cri-containerd.image=managed
Then I just use install the helm chart as normal
microk8s helm install -f values.yml cilium /var/tmp/microk8s/helm_charts/cilium -n kube-system
what goes well, but then the cilium pod fails to be created as:
Warning ErrImageNeverPull 1s (x5 over 46s) kubelet Container image "quay.io/cilium/cilium:v1.14.2@sha256:6263f3a3d5d63b267b538298dbeb5ae87da3efacf09a2c620446c873ba807d35" is not present with pull policy of Never
Warning Failed 1s (x5 over 46s) kubelet Error: ErrImageNeverPull
I have double checked the image name and digest, they match between what is loaded on the node and what the error shows. I have set the image pull policy to never as this is an offline environment and I do not have a local registry. While it might be possible to spin up the built in microk8s registry for bootstrapping I am not sure if it would even work as I am changing CNIs. I am aware that on microk8s Cilium could be deployed via addons, but this just templates and applies an older helm chart and makes upgradeability a question. I very much prefer managing this via helm.
Still the main issue is that the image is simply ignored even though it is present, I am quite lost in this issue. Appreciate any help or ideas.
Reinstalling of microk8s actually fixed the issue, upgraded from 1.27.5 to 1.28.1 and images are loaded without any problem.