gcloud builds submit does not update file

Question

When I run the command, gcloud builds submit --tag gcr.io/[MY_TAG] it builds the image and returns a success message. However, when I run docker run gcr.io/[MY_TAG] I get an error on a line of a file that did exist before but I removed it and replaced it with something else.

I'd appreciate it if you give me some ideas on possible scenarios that gcloud builds submit may not update a file and how to force it to update the file in the image.

Note that I do not have .gcloudignore anywhere in my folder, the file that is not updated does not exist in .gitignore, and I'm not able to use --no-cache because it says:

ERROR: (gcloud.builds.submit) Invalid value for [no-cache]: Cannot specify --no-cache if builds/use_kaniko property is False

I checked the last log file generated in gcloud\logs\ and the file that I need to update is indicated as "Added"!

Answer 1

It may be that you're depending on the latest tag to disambiguate image versions but this does not work as expected.

When you docker run ... ${IMAGE}:latest on your host, if the tag exists locally, whatever image has the tag, will be run. It is possible that the GCR :latest is being updated but you're not using that when you run what your host has as :latest.

Tags are a flawed mechanism for identifying container images. Tags are commonly used to denote versions|releases but they are, in fact, arbitrary labels and afford little by way of any guarantee.

You're much better off using digests as these are unique (for unique images)

gcloud builds submit \
--tag=gcr.io/${PROJECT}/test \
--project=${PROJECT}
...
latest: digest: 
sha256:e5773d45cc5dce21d78f276f87928ab47373b65ef10786722a38b22918e69d7e size: 524
DONE

gcloud container images list-tags gcr.io/${PROJECT}/test
DIGEST        TAGS    TIMESTAMP
e5773d45cc5d  latest  2020-02-06T17:21:27

If we submit the build again, the :latest tag will jump to the new image:

gcloud builds submit \
--tag=gcr.io/${PROJECT}/test \
--project=${PROJECT}
...
latest: digest: 
sha256:ab9fc60674a0e854e553baa06666c898426baec9d8d2ead0ff8513dbab960d2c size: 524
DONE

gcloud container images list-tags gcr.io/${PROJECT}/test
DIGEST        TAGS    TIMESTAMP
ab9fc60674a0  latest  2020-02-06T17:23:48
e5773d45cc5d          2020-02-06T17:21:27

You may run a container from a specific image digest this way:

docker run \
--rm --interactive --tty \
gcr.io/${PROJECT}/test@sha256:e5773d45cc5dce21d78f276f87928ab47373b65ef10786722a38b22918e69d7e

docker run \
--rm --interactive --tty \
gcr.io/${PROJECT}/test@sha256:ab9fc60674a0e854e553baa06666c898426baec9d8d2ead0ff8513dbab960d2c

An alternative approach that depends upon you using distinct tags (and is not as good as using virtually-guaranteed-to-be-distinct digests) is:

gcloud builds submit --tag=gcr.io/${PROJECT}/test:v3 --project=${PROJECT}
gcloud builds submit --tag=gcr.io/${PROJECT}/test:v4 --project=${PROJECT}

gcloud container images list-tags gcr.io/${PROJECT}/test
DIGEST        TAGS    TIMESTAMP
5946f2e093af  v4      2020-02-06T17:28:36
f03c9609c101  v3      2020-02-06T17:28:17
ab9fc60674a0  latest  2020-02-06T17:23:48
e5773d45cc5d          2020-02-06T17:21:27

NB Because we've been judicious, we have distinct digests for distinct tags. However, if we were to inadvertently reuse v4, different copies of the image would once again be inconsistent:

Let's pull the image that currently (!) corresponds to :v4:

docker pull gcr.io/${PROJECT}/test:v4
Digest: sha256:5946f2e093afbe203691cdc8a3e592db7b3e1d010619c4c97c3084de299f4d06

docker image ls
REPOSITORY                               TAG      IMAGE ID
gcr.io/dazwilkin-200206-60102073/test    v4       a1bb9204b417

Build the image reusing (!) :v4 and pull it again:

gcloud builds submit --tag=gcr.io/${PROJECT}/test:v4 --project=${PROJECT}
Digest:
sha256:e128e4fe3d61fee0c995e9ac2cd1f05635fbdc017f32eb3116fdb593826674cc

gcloud container images list-tags gcr.io/${PROJECT}/test
DIGEST        TAGS    TIMESTAMP
e128e4fe3d61  v4      2020-02-06T17:36:59
5946f2e093af          2020-02-06T17:28:36
f03c9609c101  v3      2020-02-06T17:28:17
ab9fc60674a0  latest  2020-02-06T17:23:48
e5773d45cc5d          2020-02-06T17:21:27

NB How the :v4 tag has jumped to the new digest (e128...).

But, the image with :v4 tag locally is the old image (5946...).

It's only when we (remember to) pull the :v4 tagged image again that we'll pull what's currently tagged as :v4 on GCR.

docker pull gcr.io/${PROJECT}/test:v4
Digest: sha256:e128e4fe3d61fee0c995e9ac2cd1f05635fbdc017f32eb3116fdb593826674cc
Status: Downloaded newer image for gcr.io/dazwilkin-200206-60102073/test:v4

docker image ls
REPOSITORY                                TAG      IMAGE ID
gcr.io/dazwilkin-200206-60102073/test     v4       6a916f95586a 

NB Same tag (:v4) but the digests (a hash) are unique to image builds.