Sending big amount of bytes from Swift to Python with types and @_cdecl

Question

I have two simple functions, one pass to another array of UInts. When I pass small array of 20 UInts function works, but when I pass 21576 Uints function returns small amount of bites, why is it happened?

I checked UnsafeMutablePointer<UInt8> inside have correct numbers, but on Python side they are lost.

Swift:

@_cdecl("getPointer")
public func getPointer() -> UnsafeMutablePointer<UInt8>{
    let arr: Array<UInt8> =[1,2,3.....] //here is big array
    if let buffer = buffer {
        buffer.deallocate()
        buffer.deinitialize(count: arr.count)
    }
    buffer = UnsafeMutablePointer<UInt8>.allocate(capacity: arr.count * MemoryLayout<UInt8>.stride)
    buffer!.initialize(from: arr, count: arr.count) 
 
    return buffer!
}

Python:

native_lib = ctypes.CDLL('./libH264_decoder')
native_lib.getPointer.restype = ndpointer(dtype=ctypes.c_uint8)

cont = cast(native_lib.getPointer(), c_char_p).value

returns b'\x1cri\x1aVL\xa4q\xfc\xa7\xaezb\x83HC\x94\xb4#\xde?x\xdb\xb1\xd3\x1d\x07\xb5@\xc8\x85\x0eP\xaa\x9ew\x03\x93\xfe8\xa6\x97D\xca\xc6\xcc'

Answer 1

I don't know Swift, but to return a data buffer containing nulls to ctypes you need to know the size of the buffer and can't use c_char_p as the return type since ctypes assumes null-terminated data and converts that specific type to a bytes object. Use POINTER(c_char) instead for arbitrary data that can contain nulls.

Below I've made a simple C DLL that returns a pointer to some data and returns the size in an additional output parameter. The same technique should work for Swift assuming it uses the standard C ABI to export functions, but you will need to pass back both a pointer and a size if the size is variable.

test.c

__declspec(dllexport)
char* get_data(int* size) {
    *size = 8;
    return "\x11\x22\x00\x33\x44\x00\x55\x66";
}

test.py

import ctypes as ct

dll = ct.CDLL('./test')
dll.get_data.argtypes = ct.POINTER(ct.c_int),  
dll.get_data.restype = ct.POINTER(ct.c_char)   # do NOT use ct.c_char_p

size = ct.c_int()  # allocate ctypes storage for the output parameter.
buf = dll.get_data(ct.byref(size))  # pass by reference.
print(buf[:size.value].hex(' '))    # Use string slicing to control the size.
                                    # .hex(' ') for pretty-printing the data.

Output:

11 22 00 33 44 00 55 66

Answer 2

On the python caller site, you cast the returned array to a "pointer of chars", which expects to be NULL-terminated. If you do not have any binary zeros in your array, you could check the follwing (only the important parts here):

    var arr: Array<UInt8> = [] //here is big array
    
    for _ in 0..<100 {
        for i in 0..<26 {
            arr.append(UInt8(65+i)) // A...Z
        }
    }
    arr.append(0) // Terminate C-String alike with binary Zero

    buffer = UnsafeMutablePointer<UInt8>.allocate(capacity: arr.count * MemoryLayout<UInt8>.stride)
    buffer!.initialize(from: arr, count: arr.count) 
 
    return buffer!