Is it possible to use fastapi with gremlin-python?

Question

I am trying to use fastapi with gremlin-python. That library uses tornado as transport for connecting to gremlin server, especially it uses httpclient and websockets along with tornado's ioloop.

Problem is that python prohibits nested event loops and I get runtime errors: https://pastebin.com/mcFhuKks Currently as workaround I use nest_asyncio, but that's a very unreliable solution.

Since previously I saw several issues here regarding asyncio loops (though mostly regarding aiohttp), I am curious if there is some simple way to solve that problem (e.g. somehow reuse parent event-loop).

Also I'd like to notice that gremlin-python DB connection constructor supports passing custom transport factory (instead of default), and I tried to use something like asyncio.get_event_loop() and asyncio.run_until_complete() instead of tornado's alternatives, but with no luck.

Hence I'm here for any ideas or probable solutions.

• FastAPI Version: ==0.61.1
• Python version: python:3.8-slim

Answer 1

Just a bad architecture of gremlin library. Created synchronous transport factory using websocket-client, now works fine.

Answer 2

tl;dr The nest_asyncio workaround is now used by gremlinpython itself.

---

Partial answer, hopefully useful for those that come later.

With the release of version 3.5.0, gremlin-python (aka gremlinpython) has dropped tornado in favor of AIOHTTP.

From the upgrade docs:

Python Transport Layer

With the removal of Python 2.x support the transport layer of gremlin-python has been rewritten to use a library that utilizes the asyncio event loop of Python 3. AIOHTTP utilizes Python 3’s event loop with a minimal HTTP abstraction and is now used for the transport layer. From a user’s perspective there is not much of a change except there is now new configuration options available through named parameters, see AIOHTTP ws_connect for more details. This change fixed a number of issues that were related to the IOLoop of the old Tornado transport layer, which has been completely removed from the library. An additional config which enables the driver to be used from within an event loop has been added and can be used by setting call_from_event_loop=True when connecting.

I was intrigued by the last line, so I looked into call_from_event_loop. Under the hood, it is just using nest_asyncio. Essentially, it is a more ergonomic way to achieve the nest_asyncio workaround mentioned already on this question.

Answer 3

I've released a patch to aiogremlin that solves a related issue that prevents it from being used in FastAPI, the fix can be found here: https://github.com/jerlendds/gremlinpy which is based from aiogremlin==3.3.4

In gremlin-python they create an event loop which is what is causing your error as you noticed:

def __init__(self, call_from_event_loop=None, read_timeout=None, write_timeout=None, **kwargs):
    if call_from_event_loop is not None and call_from_event_loop and not AiohttpTransport.nest_asyncio_applied:
        """
            The AiohttpTransport implementation uses the asyncio event loop. Because of this, it cannot be called
            within an event loop without nest_asyncio. If the code is ever refactored so that it can be called
            within an event loop this import and call can be removed. Without this, applications which use the
            event loop to call gremlin-python (such as Jupyter) will not work.
        """
        import nest_asyncio
        nest_asyncio.apply()
        AiohttpTransport.nest_asyncio_applied = True

    # Start event loop and initialize websocket and client to None
    self._loop = asyncio.new_event_loop()
    self._websocket = None
    self._client_session = None

aiogremlin fixes this issue, and I fix an issue in aiogremlins misuse of asyncio features