I spin up a new node in my GCP managed Kubernetes cluster. Upon startup of the new node, all the pods failed to download their image from our Artifactory instance due to a 404 layer not found error. For troubleshooting sake, we ssh'd on to the node and tried to pull the image with docker and ctr. Both of which pulled the image no problem. However, Google enforces the use of crictl to pull images so we can't just switch over to one of those.
For debugging purposes we found that the request for a crictl pull is received by Artifactory, it searches through our virtual repo, find the image layer, then continues searching even though it found the layer in a previous local repo. After the search is finished, Artifactory returns a 404. This behavior only occurs with crictl. Likewise, if we specify the local repository instead of the virtual one it will pull with no problem.
Has anyone encountered this error before? How should I mitigate this?