Deadline Exceeded Errors continuously occuring in the google cloud logging library

Question

We have recently migrated our application from Python 2 to Python 3 on Google App Engine and are encountering a recurring issue related to logging. Despite the application's functionality appearing unaffected, our error console consistently reports problems with submitting logs. This issue seems to be happening in the background but it is causing concerns regarding the reliability and integrity of our logging process, which is critical to us for analytics and security.

Error Details:

The error we are seeing is as follows:

"Failed to submit 10 logs"

And the traceback includes:

Failed to submit 10 logs
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/grpc/_channel.py", line 1160, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-02-15T04:49:06.644899178+00:00"}"

Then it further says that the above exception was the result of the following exception:

File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

We initialise our cloud logging client in the following way:

from google.cloud import logging as cloud_logging

client = cloud_logging.Client()
client.setup_logging()

We reduced the number of logs we were printing massively and it did reduce the frequency of the above error, however we are still facing this error about once a week or so. It does not impact our work flow, however we would like to stop this error from occurring permanently. Is there a way to prevent/suppress this error in any way?