is it possible to release the GIL before a C function that blocks and might call back into Python?

Question

I am wrapping a C function which performs a blocking operation (select) and then handles incoming messages. My understanding is that when a C function is going to block, the correct way to call it while allowing other threads to run is:

Py_BEGIN_ALLOW_THREADS                                                  
blocking_function();
Py_END_ALLOW_THREADS

However, it happens that this function takes as a parameter a callback pointer. This callback is called on handling the incoming message that is pre-processed by the C function. I have successfully wrapped this callback in a function which calls PyEval_CallObject(), allowing me to pass it a Python callback.

Now that I'm adding threading support, I'm wondering if it's possible to simultaneously:

• Release the GIL before calling this blocking operation.
• Have this blocking operation safely call back into the python interpreter.

Is this going to cause problems? If so, is there a way around it?

Thanks.

Answer 1

I used these API functions several months ago, and my recollection is a bit hazy, but I believe this code will solve your problem. I am assuming version 2.x (3.x may be different):

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Make your call to PyEval_CallObject() here (and any other PY API calls). */

PyGILState_Release(gstate);

(The above was taken from: Python C/API docs)

This is basically the inverse of the Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS macros. Within those, you release the GIL, but within the PyGILState functions you acquire the GIL.

Answer 2

Works if I use a global to store the result of PyEval_SaveThread, and use this in the callback to call PyEval_RestoreThread. I just need to figure out how to more cleanly pass this to the callback in my code through the callback's context pointer.