PyArango holds persistent connection with pod even after Node Death (Kubernetes)

Question

I have a Kubernetes Multinode system set up 3 three nodes. I am creating a connection between a pod on Node 2 to the Arango deployment using PyArango, the Arango Deployment has two coordinator pods one on Node 2 and one on Node 3.

I'm testing out how resilient the system is and I've noticed an issue. It seems that if I'm updating collections on Arango and my program (running on Node 2) connects to the Arango Coordinator pod on Node 3 and I power off Node 3, the connection will not time out, it will simply stay put for as long as 20 minutes.

I want the connection to timeout if the connection is idle or getting no response after 30 seconds.

I've tried some different things using the PyArango methods and no luck. How do I get python or PyArango to timeout on a stale connection asap?

At the minute this is my a snippet of the connection settings code:

 retry_policy = Retry(total=0, connect=0, read=0 ,
                       other=0, backoff_factor=0)
  while conn == None:
    try:
        conn = Connection(arango_url, username, password,
                          max_retries=retry_policy)
        conn.session.session.headers['Retry-After'] = '10'
        conn.session.session.headers['Keep-Alive'] = 'timeout=5'
      else:
        conn = Connection(arangoURL=arango_url, max_retries=retry_policy)

        conn.session.session.headers['Retry-After'] = '10'
        conn.session.session.headers['Keep-Alive'] = 'timeout=5'

Any help would be great!

Answer 1

In the same file where I make the conn = Connection(......) call. I needed to add a monkey patch to patch the AikidoSession class that PyArango uses to make the raw HTTP connections.

I added the following to the beginning of my file and now my connection times out correctly and doesn't hang for ages.

from pyArango.connection import AikidoSession

AikidoSession.Holder._base_call = AikidoSession.Holder.__call__
def _monkey_patch_call(self, *args, **kwargs):
  kwargs["timeout"] = 10
  return self._base_call(*args, **kwargs)

AikidoSession.Holder.__call__ = _monkey_patch_call

I think in my question where I was adding the timeouts, I was simply amending the local Connection object but the connection was already made so the changes never got applied to the connection.

Answer 2

You could always add connecting string to your Connection String: connect timeout=180;

This Connection Timeout is for the amount of time it takes to resolve the initial connection to the database. You can refer to this in SQL Connection timeout property doc

Or else refer to this SO. You can increase the HTTP client's timeout by using a custom HTTP client for Arango. The default is set here to 60 seconds.