Unknown error from gcp secret manager (bad gateway)

Question

I've made a cloud function that connect to secret manager, get some secrets to connect to another DB, do some stuff... every 3minutes. It works perfectly for months...

Since 2 hours, I've got several time this error I haven't seen before (while we didn't change anything)... If someone can help me on that. It seems to be related with gRPC of secret manager...

Here is the error :

·Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "502:Bad Gateway"
    debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"502:Bad Gateway", grpc_status:14, created_time:"2023-11-08T18:19:37.079022828+00:00"}"
>
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/retry.py", line 204, in retry_target
return target()
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
return func(*args, **kwargs)
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 502:Bad Gateway

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/main.py", line 45, in handler
result = get_message(event, context, temp_flow_id)
File "/workspace/main.py", line 228, in get_message
write_bqcache(bqcache_query, mode_test)
File "/workspace/main.py", line 245, in write_bqcache
BQCACHE_LOGIN, BQCACHE_PASSWORD, BQCACHE_IP = ssktools.get_authentication("BQCACHE_LOGIN_PASSWORD_IP").split(
File "/workspace/ssktools.py", line 108, in get_authentication
return __access_secret_version(PROJECT_ID, secret_id, "latest")
File "/workspace/ssktools.py", line 80, in __access_secret_version
response = client.access_secret_version(request={"name": name})
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1518, in access_secret_version
response = rpc(
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
return wrapped_func(*args, **kwargs)
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/retry.py", line 366, in retry_wrapped_func
return retry_target(
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/retry.py", line 220, in retry_target
raise exceptions.RetryError(
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 502:Bad Gateway

And for more information, ssktools.get_authentication in a shortcut for :

def __access_secret_version(project_id: str, secret_id: str, version_id: str) -> str:
"""
Access the payload for the given secret version if one exists. The version
can be a version number as a string (e.g. "5") or an alias (e.g. "latest").

Args:
    project_id (str): L'id du projet
    secret_id (str): Le secret choisi
    version_id (str): Le numéro de version

Returns:
    string: payload
"""
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()

# Build the resource name of the secret version.
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"

# Access the secret version.
response = client.access_secret_version(request={"name": name})

payload = response.payload.data.decode("UTF-8")  # type: ignore

return payload

Answer 1

As mentionned by kelso, it was a worldwide incident... https://status.cloud.google.com/incidents/SAKLsoUFXmYay2SvpNsq#xeHYqZMQgAtvK9LSJ9pP