I am having some issues running a Tensorflow model with a uvloop based server (Sanic) even though it works fine with Flask. I think it has something to do with how Sanic spawns multiple processes? I say this because if I run Sanic with 1 worker (i.e., process) then the model gets served just fine. Although I do find it perplexing that I have deployed this same model with Flask+gunicorn using multiple workers and threads per worker.
The specific error I get is:
2019-10-22 18:05:39.062833: E tensorflow/core/grappler/clusters/utils.cc:87] Failed to get device properties, error code: 3
2019-10-22 18:05:39.731232: F tensorflow/stream_executor/cuda/cuda_driver.cc:175] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: initialization error
I feel it’s at least partially related to https://github.com/pytorch/pytorch/issues/2517
I've tried using sess.as_default()and graph.as_default(), before I make the call to the code that returns the prediction to no avail. Any help would be appreciated, even just some ideas to try since I've hit a wall and am kind of lost on how to proceed.