Airflow/Buildkite ValueError: Invalid header value b'Bearer xxxx\n'

Question

We are using Airflow to run some ML training jobs, and this creates a buildkite build that generates a GitHub PR to use a new ML model. It had been working before, but now we are getting a strange error:

ValueError: Invalid header value b'Bearer xxxx\n'

I replaced the bearer token/secret with xxxx, but it's a longer string of letters/numbers. We haven't changed anything that I think should be effecting this, but it looks like Airflow has released a new version in the past few days that may be the root cause here. We are doing something like this:

from pybuildkite.buildkite import Buildkite

buildkite = Buildkite()
buildkite.set_access_token(os.getenv("BUILDKITE_API_KEY"))

resp = buildkite.builds().create_build(...)

We are using the latest versions of pybuildkite, requests, and urllib3. The full traceback looks like this:

[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     resp = buildkite.builds().create_build(
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/pybuildkite/builds.py", line 305, in create_build
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     return self.client.post(
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/pybuildkite/client.py", line 147, in post
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     return self.request(
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/pybuildkite/client.py", line 73, in request
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     response = requests.request(
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     return session.request(method=method, url=url, **kwargs)
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     resp = self.send(prep, **send_kwargs)
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     r = adapter.send(request, **kwargs)
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     resp = conn.urlopen(
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     httplib_response = self._make_request(
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     conn.request(method, url, **httplib_request_kw)
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/http/client.py", line 1230, in request
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     self._send_request(method, url, body, headers, encode_chunked)
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/http/client.py", line 1271, in _send_request
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     self.putheader(hdr, value)
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 224, in putheader
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     _HTTPConnection.putheader(self, header, *values)
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -   File "/usr/local/lib/python3.8/http/client.py", line 1208, in putheader
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO -     raise ValueError('Invalid header value %r' % (values[i],))
[2022-05-25, 12:16:58 UTC] {pod_manager.py:197} INFO - ValueError: Invalid header value b'Bearer xxxx\n'

I'm finding it hard to trace down though the Airflow code what's happening, but we are using a secret and passing this to a kubernetes pod in our airflow code:

from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
    KubernetesPodOperator,
)
from kubernetes.client import models as k8s
from airflow.kubernetes.secret import Secret

secret_buildkite = Secret(
    deploy_type="env",
    deploy_target="BUILDKITE_API_KEY",
    secret="enrichment-ml-env-buildkite-api-key",
    key="buildkite-api-key",
)

model_retraining = KubernetesPodOperator(
            namespace="airflow",
            image="gcr.io/...",  # container name changed to ... for SO post
            image_pull_policy="Always",
            arguments=["--environment", target_cluster, "--market", market, "--income"],
            secrets=secrets,
            ...) # there are more args but I'm not sure they're relevant

Anyone have ideas what could be going wrong or how to fix it?