Pickling a C extension

Question

related to this question python: pickling c objects I extended my Python C extension with a __reduce__ function:

static PyMethodDef Kraken_methods[] = {                                                                
    {"name", (PyCFunction)Kraken_name, METH_NOARGS, "Return the first and last name"},
    {"__reduce__", (PyCFunction)Kraken_reduce, METH_NOARGS, "Necessary for pickling objects"},
    {NULL}  /* Sentinel */                                                                              
}; 

My __reduce__ function (Kraken_reduce) looks, as the first try, like the following:

static PyObject *Kraken_reduce(Kraken* self){               

char* x = "hello from reduce\n";
printf("reduce called !!!\n");
return x; 
}

Of course I would expect a kind of an exception or something caused by returning the wrong type. But when I call the function explicit

kr_mod = kr_module.kr_module(api_key, sec_key)
kr_mod.__reduce__()

I get the following error:

  File "pipe_multiprocessor.py", line 99, in <module>
    kr_mod.__reduce__()
  File "/home/stephan/anaconda3/lib/python3.6/copyreg.py", line 65, in _reduce_ex
    raise TypeError("can't pickle %s objects" % base.__name__)
TypeError: can't pickle kr_module objects

The printf instructions are not executed.

On the other hand, if I import the C extension in python and define the __reduce__ method after the import:

class kraken_mod(kr_module.kr_module):
    def __reduce__(self):
            return (self.__class__, (api_key, sec_key, ))

Pickling (for the purpose of multiprocessing) works fine!

So what I'm doing wrong? I would like to define the __reduce__ method in the C code so that I can omit the additional in-python definition.

EDIT: After I found the error in my toolchain which causes the method not compile and execute, I'm not getting any further in the definition of the __reduce__ method.

static PyObject *Kraken_reduce(kr_module* self){

    PyObject *getattr;
    PyObject *builtins;
    PyObject *tuple;

    _Py_IDENTIFIER(getattr);
    builtins = PyEval_GetBuiltins();

    getattr = _PyDict_GetItemId(builtins, &PyId_getattr);
    PyObject_Print(getattr, stdout, 0);

    printf("reduce called !!!\n");

    tuple = Py_BuildValue("O(ss)", getattr, "123", "456");
    return tuple;
}

The __reduce__ method is now called when the kr_module is send to another process. In the new process an exception is thrown: (I pulled the exception back in the main thread)

Exception in target_0(): <class 'AttributeError'>
Exception in target_0(): 'str' object has no attribute '456'

Can anyone help me?

Answer 1

I found a solution for my problem: The following function works for me:

 static PyObject *Kraken_reduce(kr_module* self){

    PyObject *tuple;
    PyObject *obj;
    PyObject *attr;

    obj = (PyObject*)self;

    attr = PyObject_GetAttrString(obj, "__class__");

    tuple = Py_BuildValue("O(ss)", attr, self->kr_api->s_api_key, self->kr_api->s_sec_key);

    return tuple;
}

The complete soruce for the file is available here at Github