Accessing a c++ vector/array from python via cython

1k views Asked by At

I am wrapping a c++ program via cython and would like to make a vector from this program available in Python.

I have written a getter method for the vector, which works fine except that it returns a copy of the data:

# in cython
from libcpp.vector cimport vector
cimport numpy as np

cdef np.ndarray[double] vector_to_double_arr(vector[double] vec):
    cdef double[::1] arr = <double [:vec.size()]>vec.data()
    return np.asarray(arr)

def get_variable()
    return vector_to_double_arr(my_c_vector) # my_c_vector is a global variable defined elsewhere

That is, in python, an assignment to the returned array has no effect, since the value is assigned to the copy only.

# in python

get_variable()[0] = 4 # has no effect

In the past, I have circumvented this problem by creating numpy arrays first and operating in c++ on the internal data. However, now I have to deal with a pre-existing vector.

I have thought of returning the memoryview directly:

cdef np.ndarray[double] vector_to_double_arr(vector[double] vec):
    cdef double[::1] arr = <double [:vec.size()]>vec.data()
    return arr 

However, I have not figured out how to make the memoryview appear as an array in Python. I would need some object that wraps this memoryview and can write to it in a proper way. I was thinking of creating a numpy array and assigning to the np.ndarray.data property, but writing to it is bad practice and did not work.

How could I write to the c++ vector from Python? As an extension: how would I do this, if I were not dealing with a vector but a double array?

Edit

I have now implemented the solution suggested by @ead by copying the code:

from libc.stdlib cimport free
from cpython.object cimport PyObject
from cpython.ref cimport Py_INCREF
np.import_array()


cdef class MemoryNanny:
    cdef void* ptr # set to NULL by "constructor"
    def __dealloc__(self):
        print("freeing ptr=", <unsigned long long>(self.ptr)) #just for debugging
        free(self.ptr)
        
    @staticmethod
    cdef create(void* ptr):
        cdef MemoryNanny result = MemoryNanny()
        result.ptr = ptr
        print("nanny for ptr=", <unsigned long long>(result.ptr)) #just for debugging
        return result
    
cdef extern from "numpy/arrayobject.h":
    # a little bit awkward: the reference to obj will be stolen
    # using PyObject*  to signal that Cython cannot handle it automatically
    int PyArray_SetBaseObject(np.ndarray arr, PyObject *obj) except -1 # -1 means there was an error
          
cdef array_from_ptr(void * ptr, np.npy_intp N, int np_type):
    cdef np.ndarray arr = np.PyArray_SimpleNewFromData(1, &N, np_type, ptr)
    nanny = MemoryNanny.create(ptr)
    Py_INCREF(nanny) # a reference will get stolen, so prepare nanny
    PyArray_SetBaseObject(arr, <PyObject*>nanny) 
    return arr

I access the pointer then as follows:

cdef MyClass()
    ...
    @property
    def myArray(self):
        return array_from_ptr(myArrayPointer, myArrayPointerLength, np.NPY_FLOAT64)

Accessing the property from Python works well, but as soon as I change any value in the array (which works, too) the program halts.

m = MyClass()
print(m.myArray) # prints what it is supposed to print
m.myArray[0] = 1 # works
print(m.myArray) # prints what is expected, but terminates the program after freeing the freeing the pointer.
...

When I delete the last print statement in the code block above, everything works still. Why? I am puzzled. What could I do to solve the problem?

0

There are 0 answers