Can not pull any image in minikube

Question

Im on macOS and im using minikube with hyperkit driver: minikube start --driver=hyperkit

and everything seems ok...

with minikube status:

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

with minikube version:

minikube version: v1.24.0

with kubectl version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

and with kubectl get no:

NAME       STATUS   ROLES                  AGE   VERSION
minikube   Ready    control-plane,master   13m   v1.22.3

my problem is when i deploy anything, it wont pull any image...

for instance:

kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.4

then kubectl get pods:

NAME                              READY   STATUS             RESTARTS   AGE
hello-minikube-6ddfcc9757-nfc64   0/1     ImagePullBackOff   0          13m

then i tried to figure out what is the problem?

k describe pod/hello-minikube-6ddfcc9757-nfc64

here is the result:

Name:         hello-minikube-6ddfcc9757-nfc64
Namespace:    default
Priority:     0
Node:         minikube/192.168.64.8
Start Time:   Sun, 16 Jan 2022 10:49:27 +0330
Labels:       app=hello-minikube
              pod-template-hash=6ddfcc9757
Annotations:  <none>
Status:       Pending
IP:           172.17.0.5
IPs:
  IP:           172.17.0.5
Controlled By:  ReplicaSet/hello-minikube-6ddfcc9757
Containers:
  echoserver:
    Container ID:   
    Image:          k8s.gcr.io/echoserver:1.4
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k5qql (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-k5qql:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  18m                   default-scheduler  Successfully assigned default/hello-minikube-6ddfcc9757-nfc64 to minikube
  Normal   Pulling    16m (x4 over 18m)     kubelet            Pulling image "k8s.gcr.io/echoserver:1.4"
  Warning  Failed     16m (x4 over 18m)     kubelet            Failed to pull image "k8s.gcr.io/echoserver:1.4": rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     16m (x4 over 18m)     kubelet            Error: ErrImagePull
  Warning  Failed     15m (x6 over 18m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    3m34s (x59 over 18m)  kubelet            Back-off pulling image "k8s.gcr.io/echoserver:1.4"

then tried to get some logs!:

k logs pod/hello-minikube-6ddfcc9757-nfc64 and k logs deploy/hello-minikube

both returns the same result:

Error from server (BadRequest): container "echoserver" in pod "hello-minikube-6ddfcc9757-nfc64" is waiting to start: trying and failing to pull image

this deployment was an example from minikube documentation

but i have no idea why it doesnt pull any image...

Answer 1

I had exactly same problem. I found out that my internet connection was slow, the timout to pull an image is 120 seconds, so kubectl could not pull the image in under 120 seconds.

first use minikube to pull the image you need for example:

minikube image load k8s.gcr.io/echoserver:1.4

and then everything will work because now kubectl will use the image that is stored locally.

Answer 2

I encountered some similar issue, it is fixed by using echoserver:1.10 instead of echoserver:1.4

Answer 3

According to this article:

The status ImagePullBackOff means that a Pod couldn’t start, because Kubernetes couldn’t pull a container image. The ‘BackOff’ part means that Kubernetes will keep trying to pull the image, with an increasing delay (‘back-off’).

Here is also a handbook about pushing images into a minikube cluster.

This handbook describes your issue:

Unable to pull images..Client.Timeout exceeded while awaiting headers

Unable to pull images, which may be OK:

failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.3": output: Error response from daemon:
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection
(Client.Timeout exceeded while awaiting headers)

This error indicates that the container runtime running within the VM does not have access to the internet.

See possible workarounds.