Authentication methods using a JSON key file: unauthorized: GCR login failed

Question

Mostly addressed to: google-cloud-platform

Overall problem I am trying to solve is; to pull images from Google Container Registry from private Kubernetes.

Update Just added heptio-contour if some one over there have come across this - as the good people at Heptio has created the script mentioned in the question further down - thanks.

First step is to just use the Service Account with a JSON key - as described here.
But when I run:

cat gcr-sa-key.json | docker login -u _json_key --password-stdin https://gcr.io

I should be able to login docker, but it fails with:

cat gcr-sa-key.json | docker login -u _json_key --password-stdin https://gcr.io
Error response from daemon: Get https://gcr.io/v2/: unauthorized: GCR login failed. You may have invalid credentials. To login successfully, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Note: I got the gcr-sa-key.json file from running this - keep in mind that I am overall trying to use this from Kubernetes.

I expect this to be a Google issue, but/and if I do run as described in the doc from Heptio I get:

Events:
  Type     Reason          Age                From                                        Message
  ----     ------          ----               ----                                        -------
  Normal   Scheduled       50s                default-scheduler                           Successfully assigned default/<image-name>-deployment-v1-844568c768-5b2rt to my-cluster-digitalocean-1-7781
  Normal   Pulling         37s (x2 over 48s)  kubelet, my-cluster-digitalocean-1-7781  pulling image "gcr.io/<project-name><image-name>:v1"
  Warning  Failed          37s (x2 over 48s)  kubelet, my-cluster-digitalocean-1-7781  Failed to pull image "gcr.io/<project-name>/<image-name>:v1": rpc error: code = Unknown desc = Error response from daemon: pull access denied for gcr.io/<project-name>/<image-name>, repository does not exist or may require 'docker login'
  Warning  Failed          37s (x2 over 48s)  kubelet, my-cluster-digitalocean-1-7781  Error: ErrImagePull
  Normal   SandboxChanged  31s (x7 over 47s)  kubelet, my-cluster-digitalocean-1-7781  Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         29s (x6 over 45s)  kubelet, my-cluster-digitalocean-1-7781  Back-off pulling image "gcr.io/<project-name>/<image-name>:v1"
  Warning  Failed          29s (x6 over 45s)  kubelet, my-cluster-digitalocean-1-7781  Error: ImagePullBackOff

Just info. that might be related, I saw this issue on github.

Answer 1

You are missing the most important bit, you need to somehow grant a Kubernetes' default service account (the simplest approach) the permission to access your private container registry while pulling images. You do this in three steps:

1. Create and grant your GCP service account appropriate role in AIM (at least Storage Object Viewer) as explain here in official doc
2. Create kubernetes secret (of 'docker-registry' type) using downloaded JSON key for your GCP service account

kubectl create secret docker-registry my-private-gcr-readonly \
--docker-server=gcr.io \
--docker-username=_json_key \
--docker-password="$(cat /usr/local/home/demo/414141.json)" \
[email protected]

3. Grant your default Kubernetes service account (your PODs are running under its security context by default) the right to pull Image from private GCR repo. This is done indirectly, by assigning it the secret for imagePull operation:

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "my-private-gcr-readonly"}]}'

That's it !

PS.

You can also check this tutorial, that explains both ways of accessing Google Container Registry from within Kubernetes cluster (using JSON Key or Access token)