Pod deployed over GitHub Repository by JKube causes 401 while deploying in MicroK8

Question

I am trying installing a simple Spring Boot application by JKube on a microk8s cluster. As remote private docker repository I am using GitHub Packages (beta). While the microk8s is getting the pod, he unsuccessfully tries to deploy it and brings out following error:

Command:

kubectl describe pod walking-skeleton-5df7c4998c-9blmz -n walking

Output:

Name:         walking-skeleton-5df7c4998c-9blmz
Namespace:    walking
Priority:     0
Node:         server-k8s-main/MYIP
Start Time:   Tue, 16 Feb 2021 23:01:57 +0100
Labels:       app=walking-skeleton
              group=de.MYNAME
              pod-template-hash=5df7c4998c
              provider=jkube
              version=0.0.2-SNAPSHOT
Annotations:  cni.projectcalico.org/podIP: 10.1.4.29/32
              cni.projectcalico.org/podIPs: 10.1.4.29/32
              jkube.io/git-branch: main
              jkube.io/git-commit: c0ba4b997db326fc8a7018adfca4f5a4d74aea39
              jkube.io/git-url: [email protected]:MYNAME/MYPROJECT.git
              jkube.io/scm-tag: HEAD
              jkube.io/scm-url: https://github.com/spring-projects/spring-boot/walking-skeleton
Status:       Pending
IP:           10.1.4.29
IPs:
  IP:           10.1.4.29
Controlled By:  ReplicaSet/walking-skeleton-5df7c4998c
Containers:
  spring-boot:
    Container ID:   
    Image:          docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT
    Image ID:       
    Ports:          8080/TCP, 9779/TCP, 8778/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8080/actuator/health delay=180s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:8080/actuator/health delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      KUBERNETES_NAMESPACE:  walking (v1:metadata.namespace)
      HOSTNAME:              walking-skeleton-5df7c4998c-9blmz (v1:metadata.name)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jq7q9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-jq7q9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jq7q9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  13m                  default-scheduler  Successfully assigned walking/walking-skeleton-5df7c4998c-9blmz to server-k8s-main
  Normal   Pulling    11m (x4 over 13m)    kubelet            Pulling image "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT"
  Warning  Failed     11m (x4 over 13m)    kubelet            Error: ErrImagePull
  Warning  Failed     11m (x4 over 13m)    kubelet            Failed to pull image "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT": rpc error: code = Unknown desc = failed to pull and unpack image "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT": failed to resolve reference "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT": unexpected status code [manifests 0.0.2-SNAPSHOT]: **401 Unauthorized**
  Normal   BackOff    11m (x6 over 13m)    kubelet            Back-off pulling image "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT"
  Warning  Failed     3m1s (x43 over 13m)  kubelet            Error: ImagePullBackOff

I think, the interesting line is:

failed to resolve reference "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT": unexpected status code [manifests 0.0.2-SNAPSHOT]: **401 Unauthorized**

My plugin config in the pom.xml looks like:

        <profile>
            <id>kubernetes-deploy</id>
            <properties>
                <jkube.docker.registry>docker.pkg.github.com/MYNAME/MYPROJECT</jkube.docker.registry>
                <jkube.generator.name>docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:${project.version}
                </jkube.generator.name>
            </properties>

            <build>
                <plugins>
                    <plugin>
                        <groupId>org.eclipse.jkube</groupId>
                        <artifactId>kubernetes-maven-plugin</artifactId>
                        <version>1.1.0</version>
                        <configuration>
                            <namespace>${k8s.namespace}</namespace>
                            <resources>
                                <replicas>2</replicas>
                                <secrets>
                                    <secret>
                                        <dockerServerId>docker.pkg.github.com/MYNAME/MYPROJECT</dockerServerId>
                                        <name>github</name>
                                        <namespace>${k8s.namespace}</namespace>
                                    </secret>
                                </secrets>
                            </resources>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>

The namespace is configured like <k8s.namespace>walking</k8s.namespace> in the properties. And as secret in the settings:

        <server>
            <id>docker.pkg.github.com/MYNAME/MYPROJECT</id>
            <username>MYNAME</username>
            <password>nice try :-)</password>
        </server>

I also checked, that the secret has been deployed on the microk8s:

kubectl get secrets -n walking

NAME                  TYPE                                  DATA   AGE
default-token-jq7q9   kubernetes.io/service-account-token   3      37m
github                kubernetes.io/dockercfg               1      37m

Can you help me? What I am missing? Thank you in advance :-)

Answer 1

I've had the same issue when upgrading an existing cluster to K8s 1.24. The image of a deployment couldn't be loaded anymore due to an authentication issue.

The solution for me was to change the registry from docker.pkg.github.com to ghcr.io. This is mentioned in a containerd GitHub issue, which references the GitHub docs:

The domain in the pull URL will be ghcr.io instead of docker.pkg.github.com.