Calling gcloud from bazel genrule

Question

I am having some issues getting gcloud to run in a Bazel genrule. Looks like python path related issues.

genrule(
    name="foo",
    outs=["bar"],
    srcs=[":bar.enc"],
    cmd="gcloud decrypt --location=global --keyring=foo --key=bar --plaintext-file $@ --ciphertext-file $(location bar.enc)"
)

The exception is:

ImportError: No module named traceback

From:

 try:
    gcloud_main = _import_gcloud_main()
  except Exception as err:  # pylint: disable=broad-except
    # We want to catch *everything* here to display a nice message to the user
    # pylint:disable=g-import-not-at-top
    import traceback
    # We DON'T want to suggest `gcloud components reinstall` here (ex. as
    # opposed to the similar message in gcloud_main.py), as we know that no
    # commands will work.
    sys.stderr.write(
        ('ERROR: gcloud failed to load: {0}\n{1}\n\n'
         'This usually indicates corruption in your gcloud installation or '
         'problems with your Python interpreter.\n\n'
         'Please verify that the following is the path to a working Python 2.7 '
         'executable:\n'
         '    {2}\n\n'
         'If it is not, please set the CLOUDSDK_PYTHON environment variable to '
         'point to a working Python 2.7 executable.\n\n'
         'If you are still experiencing problems, please reinstall the Cloud '
         'SDK using the instructions here:\n'
         '    https://cloud.google.com/sdk/\n').format(
             err,
             '\n'.join(traceback.format_exc().splitlines()[2::2]),
             sys.executable))
    sys.exit(1)

My questions are:

• How do I best call gcloud from a genrule?
• What are the parameters needed to specify the python path?
• How is Bazel blocking this?

Update: Able to get it to run by specifying the CLOUDSDK_PYTHON.

Answer 1

Indeed, bazel runs in a sandbox, hence gcloud cannot find its dependencies. Acutally, I'm surprised gcloud can be invoked at all.

To proceed, I would wrap gcloud in a bazel py_binary and refer it with tools attribute in the genrule. You also need to wrap it with location in the cmd. In the end, you will have

genrule(
    name = "foo",
    outs = ["bar"],
    srcs = [":bar.enc"],
    cmd = "$(location //third_party/google/gcloud) decrypt --location=global --keyring=foo --key=bar --plaintext-file $@ --ciphertext-file $(location bar.enc)",
    tools = ["//third_party/google/gcloud"],
)

And for that you define in third_party/google/gcloud/BUILD (or anywhere your want, I just used a path that makes sense to me)

py_binary(
    name = "gcloud",
    srcs = ["gcloud.py"],
    main = "gcloud.py",
    visibility = ["//visibility:public"],
    deps = [
        ":gcloud_sdk",
    ],
)
py_library(
  name = "gcloud_sdk",
  srcs = glob(
      ["**/*.py"],
      exclude = ["gcloud.py"],
      # maybe exclude tests and unrelated code, too.
  ),
  deps = [
    # Whatever extra deps are needed by gcloud to compile
  ]
)

Answer 2

I had a similar issue, worked for me running this command:

export CLOUDSDK_PYTHON=/usr/bin/python

(this was answered above as an update but I felt to post the whole command for future people coming here)