Wrong result using a PyCapsule created from a method in Cython

Question

We would need to create a PyCapsule from a method of a class in Cython. We managed to write a code which compiles and even runs without error but the results are wrong.

A simple example is here: https://github.com/paugier/cython_capi/tree/master/using_cpython_pycapsule_class

The capsules are executed by Pythran (one needs to use the version on github https://github.com/serge-sans-paille/pythran).

The .pyx file:

from cpython.pycapsule cimport PyCapsule_New


cdef int twice_func(int c):
    return 2*c


cdef class Twice:
    cdef public dict __pyx_capi__

    def __init__(self):
        self.__pyx_capi__ = self.get_capi()

    cpdef get_capi(self):
        return {
            'twice_func': PyCapsule_New(
                <void *>twice_func, 'int (int)', NULL),
            'twice_cpdef': PyCapsule_New(
                <void *>self.twice_cpdef, 'int (int)', NULL),
            'twice_cdef': PyCapsule_New(
                <void *>self.twice_cdef, 'int (int)', NULL),
            'twice_static': PyCapsule_New(
                <void *>self.twice_static, 'int (int)', NULL)}

    cpdef int twice_cpdef(self, int c):
        return 2*c

    cdef int twice_cdef(self, int c):
        return 2*c

    @staticmethod
    cdef int twice_static(int c):
        return 2*c

The file compiled by pythran (call_capsule_pythran.py).

# pythran export call_capsule(int(int), int)

def call_capsule(capsule, n):
    r = capsule(n)
    return r

Once again it is a new feature of Pythran so one needs the version on github...

And the test file:

try:
    import faulthandler
    faulthandler.enable()
except ImportError:
    pass

import unittest

from twice import Twice
from call_capsule_pythran import call_capsule


class TestAll(unittest.TestCase):
    def setUp(self):
        self.obj = Twice()
        self.capi = self.obj.__pyx_capi__

    def test_pythran(self):
        value = 41
        print('\n')

        for name, capsule in self.capi.items():
            print('capsule', name)
            result = call_capsule(capsule, value)

            if name.startswith('twice'):
                if result != 2*value:
                    how = 'wrong'
                else:
                    how = 'good'

                print(how, f'result ({result})\n')


if __name__ == '__main__':
    unittest.main()

It is buggy and gives:

capsule twice_func
good result (82)

capsule twice_cpdef
wrong result (4006664390)

capsule twice_cdef
wrong result (4006664390)

capsule twice_static
good result (82)

It shows that it works fine for the standard function and for the static function but that there is a problem for the methods.

Note that the fact that it works for two capsules seems to indicate that the problem does not come from Pythran.

Edit

After DavidW's comments, I understand that we would have to create at run time (for example in get_capi) a C function with the signature int(int) from the bound method twice_cdef whose signature is actually int(Twice, int).

I don't know if this is really impossible to do with Cython...

Answer 1

To follow up/expand on my comments:

The basic issue is that the Pythran is expecting a C function pointer with the signature int f(int) to be contained within the PyCapsule. However, the signature of your methods is int(PyObject* self, int c). The 2 gets passed as self (not causing disaster since it isn't actually used...) and some arbitrary bit of memory is used in place of the int c. Unfortunately it isn't possible to use pure C code to create a C function pointer with "bound arguments" so Cython can't (and realistically won't be able to) do it.

Modification 1 is to get better compile-time type checking of what you're passing to your PyCapsules by creating a function that accepts the correct types and casting in there, rather than just casting to <void*> blindly. This doesn't solve your problem but warns you at compile-time when it isn't going to work:

ctypedef int(*f_ptr_type)(int)

cdef make_PyCapsule(f_ptr_type f, string):
    return PyCapsule_New(
                <void *>f, string, NULL)

# then in get_capi:
'twice_func': make_PyCapsule(twice_func, b'int (int)'), # etc

It is actually possible to create C function from arbitrary Python callables using ctypes (or cffi) - see Using function pointers to methods of classes without the gil (bottom of answer). This adds an extra layer of Python calls so isn't terribly quick, and the code is a bit messy. ctypes achieves this by using runtime code generation (which isn't that portable or something you can do in pure C) to build a function on the fly and then create a pointer to that.

Although you claim in the comments that you don't think you can use the Python interpreter, I don't think this is true - Pythran generates Python extension modules (so is pretty bound to the Python interpreter) and it seems to work in your test case shown here:

 _func_cache = []

cdef f_ptr_type py_to_fptr(f):
    import ctypes
    functype = ctypes.CFUNCTYPE(ctypes.c_int,ctypes.c_int)
    ctypes_f = functype(f)
    _func_cache.append(ctypes_f) # ensure references are kept
    return (<f_ptr_type*><size_t>ctypes.addressof(ctypes_f))[0]

# then in make_capi:
'twice_cpdef': make_PyCapsule(py_to_fptr(self.twice_cpdef), b'int (int)')

Unfortunately it only works for cpdef and not cdef functions since it does rely on having a Python callable. cdef functions can be made to work with a lambda (provided you change get_capi to def instead of cpdef):

'twice_cdef': make_PyCapsule(py_to_fptr(lambda x: self.twice_cdef(x)), b'int (int)'),

It's all a little messy but can be made to work.