CFFI implementation quits due to access violation (Process finished with exit code -1073741819 (0xC0000005))

Question

In comparison with Ctypes, everything being the same except for the syntax, my CFFI implementation quits throwing access violation.

Process finished with exit code -1073741819 (0xC0000005)

Environment: windows 10 (64-bit) Python versions tried:

• 3.9 (64-bit)
• 3.10 (64-bit)

I have written two separate implementations of calling a DLL function which returns a point cloud. The DLL accepts a pointer to buffer allocated for a structure and the size of the buffer.

Note: while writing the data to the pointcloud buffer, the DLL takes into consideration only the size passed in iBufSize argument, not for the nPoints variable in the structure.

from cffi import FFI

ffi = FFI()
lib = ffi.dlopen("PointCloud.dll")
ffi.cdef(
"""
typedef struct {
    double x;
    double y;
    double z;
} PointXYZ;

typedef struct {
    uint32_t nPoints;
    PointXYZ * pointcloud;
} PointCloud;

int GetPointCloud(void * ptrPointCloudBuf, int iBufSize);
"""
)

nPts = 5000000
buf_size = nPts * ffi.sizeof('PointXYZ')
pc_struct = ffi.new(
    "PointCloud *", {
        "nPoints": ffi.cast("uint32_t", 0),
        "pointcloud": ffi.new("PointXYZ []", nPts )
    }
)

retcode = lib.GetPointCloud(pc_struct, buf_size)

If the everything is fine, the GetPointCloud() function returns retcode to be SUCCESS with the data written to the pointcloud buffer, else appropriate errorcode to the `retcode'.

The same implementation with Ctypes gets executed without any problem.

from ctypes import *

lib = cdll.LoadLibrary("PointCloud.dll")

class PointXYZ(Structure):
    _fields_ = [
        ('x', c_double),
        ('y', c_double),
        ('z', c_double)
    ]

class PointCloud(Structure):
    _fields_ = [
        ('nPoints', c_uint32),
        ('pointcloud', POINTER(PointXYZ)),
    ]

GetPointCloud = lib['GetPointCloud']
GetPointCloud.restype = c_int
GetPointCloud.argtypes = [POINTER(PointCloud), c_int]

nPts = 5000000
buf_size = nPts * sizeof(PointXYZ)
pc_struct = pointer(
            PointCloud(
                nPoints=c_uint32(0),
                pointcloud=cast((PointXYZ * nPts)(), POINTER(PointXYZ))
            )
        )

retcode = lib.GetPointCloud(pc_struct, buf_size)

Note:

- DLL (pure C implementation) Source code is not availabele to me.

- Both, CFFI and Ctypes, implementations were working smoothly for previous versions of DLL, until they changed something in the DLL, while keeping the external interface same.

The contents of the header file supplied with the DLL. Of course it has a few of other functions. But I have pasted only the part of our interest.

#ifndef POINTCLOUD_H
#define POINTCLOUD_H
#if defined(_MSC_VER)
    #define POINTCLOUD_DLL_EXPORT __declspec(dllexport)        
#else
    #define POINTCLOUD_DLL_EXPORT __attribute__ ((visibility ("default")))
    #define _stdcall
#endif

// Point structs
typedef struct {
    double x;
    double y;
    double z;
} PointXYZ;

struct PointCloud {
    uint32_t number_of_points;
    PointXYZ * pointcloud;
    PointCloud() {
        pointcloud= nullptr;
        number_of_points = 0;
    }
};

extern "C" POINTCLOUD_DLL_EXPORT int  GetPointCloud(void* buffer, int buffer_size);

#endif // POINTCLOUD_H

I have access to the source code of their C/C++ demo application which uses the DLL, which runs absolutely fine. A code snippet for the respective C code is:

struct PointXYZ
{
    double x;
    double y;
    double z;
};
struct PointCloud {
    uint32_t nPoints;
    PointXYZ * pointcloud;
    PointCloud() {
        pointcloud = nullptr;
        nPoints = 0;
    }
};

PointCloud pcloud;
int nPts = 5000000;

pcloud.pointcloud = new PointXYZ[nPts];
buf_size = sizeof(PointXYZ) * nPts;
int n = GetPointCloud(&pcloud, buf_size);   
if(n == 0)
{
    printf("Success, result = %d\n", n);    
}
else
{
    printf("Failure, result = %d\n", n);
}
/*
Do the rest with the point cloud.
*/

Answer 1

I've got it, it's an invalid temporary reference resulting in a dangling pointer usage which is Undefined Behavior (common pitfall, especially for Python programmers that come in contact with C).

According to [ReadTheDocs]: Using the ffi/lib objects - Working with pointers, structures and arrays (emphasis is mine):

However, this is always wrong (usage of freed memory):

p = ffi.new("char **", ffi.new("char[]", "hello, world"))
# WRONG!  as soon as p is built, the inner ffi.new() gets freed!

I prepared a small example.

• dll00.c:

  #include <inttypes.h>
  #include <stdio.h>
  #include <stdlib.h>
  #include <time.h>
  
  #if defined(_WIN32)
  #  define DLL00_EXPORT_API __declspec(dllexport)
  #else
  #  define DLL00_EXPORT_API
  #endif
  
  
  #if defined(__cplusplus)
  extern "C" {
  #endif
  
  DLL00_EXPORT_API int GetPointCloud(void *pBuf, int bSize);
  
  #if defined(__cplusplus)
  }
  #endif
  
  
  typedef struct {
      double x;
      double y;
      double z;
  } PointXYZ;
  
  typedef struct {
      uint32_t nPoints;
      PointXYZ *pointcloud;
  } PointCloud;
  
  
  int GetPointCloud(void *pBuf, int bSize)
  {
      if ((pBuf == NULL) || (bSize <= 0)) {
          return -1;
      }
      unsigned int seed = (unsigned int)time(NULL);
      //printf("%d %d\n", RAND_MAX, seed);
      srand(seed);
      int r = rand();
      printf("From C: Attempting to return %d points\n", r);
      size_t req = r * sizeof(PointXYZ);
      if (req > bSize) {
          return req;
      }
      PointCloud *ppc = (PointCloud*)pBuf;
      for (int i = 0; i < r; ++i)
          ppc->pointcloud[i] = (PointXYZ){i * 3, i * 3 + 1, i * 3 + 2};
      ppc->nPoints = r;
      return 0;
  }
  

• code00.py:

  #!/usr/bin/env python
  
  import ctypes as cts
  #import random
  import sys
  
  
  DLL_NAME = "./PointCloud.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so")
  
  class PointXYZ(cts.Structure):
      _fields_ = (
          ("x", cts.c_double),
          ("y", cts.c_double),
          ("z", cts.c_double),
      )
  
  PPointXYZ = cts.POINTER(PointXYZ)
  
  
  class PointCloud(cts.Structure):
      _fields_ = (
          ("nPoints", cts.c_uint32),
          ("pointcloud", PPointXYZ),
      )
  
  PPointCloud = cts.POINTER(PointCloud)
  
  
  def call_dll_func(func, pc, size, pc_transform=lambda arg: arg):
      res = func(pc_transform(pc), size)
      print("\nFunction {:} returned: {:d}".format(func, res))
      if res != 0:
          return
      idxs = (0, pc.nPoints // 2, pc.nPoints - 1)
      print("\nPoints: {:d}\nData ({:d} points):".format(pc.nPoints, len(idxs)))
      for i in idxs:
          p = pc.pointcloud[i]
          print("  {:d}: {:.1f} {:.1f} {:.1f}".format(i, p.x, p.y, p.z))
  
  
  def main(*argv):
      #random.seed()
      dll = cts.CDLL(DLL_NAME)
      GetPointCloud = dll.GetPointCloud
      GetPointCloud.argtypes = ()
      GetPointCloud.restype = cts.c_int
  
      pts = 5000000
      buf_size = pts * cts.sizeof(PointXYZ)
      pc = PointCloud(nPoints=cts.c_uint32(0), pointcloud=cts.cast((PointXYZ * pts)(), PPointXYZ))
  
      call_dll_func(GetPointCloud, pc, buf_size, pc_transform=lambda arg: cts.byref(arg))
  
  
  if __name__ == "__main__":
      print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                     64 if sys.maxsize > 0x100000000 else 32, sys.platform))
      rc = main(*sys.argv[1:])
      print("\nDone.\n")
      sys.exit(rc)
  

• code01.py:

  #!/usr/bin/env python
  
  import sys
  
  from cffi import FFI
  
  from code00 import DLL_NAME, \
      call_dll_func
  
  
  ffi = FFI()
  
  ffi.cdef(
  """
  typedef struct {
      double x;
      double y;
      double z;
  } PointXYZ;
  
  typedef struct {
      uint32_t nPoints;
      PointXYZ * pointcloud;
  } PointCloud;
  
  int GetPointCloud(void *pBuf, int bSize);
  """
  )
  
  
  def main(*argv):
      dll = ffi.dlopen(DLL_NAME)
  
      pts = 5000000
      buf_size = pts * ffi.sizeof("PointXYZ")
      pc_buf = ffi.new("PointXYZ []", pts)  # @TODO - cfati: - Take it outside
      pc = ffi.new(
          "PointCloud *", {
              "nPoints": ffi.cast("uint32_t", 0),
              "pointcloud": pc_buf,
          }
      )
  
      GetPointCloud = dll.GetPointCloud
  
      call_dll_func(GetPointCloud, pc, buf_size)
  
  
  if __name__ == "__main__":
      print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                     64 if sys.maxsize > 0x100000000 else 32, sys.platform))
      rc = main(*sys.argv[1:])
      print("\nDone.\n")
      sys.exit(rc)
  

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q076053999]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2019\VC\Auxiliary\Build\vcvarsall.bat" x64 > nul

[prompt]>
[prompt]> dir /b
code00.py
code01.py
dll00.c

[prompt]>
[prompt]> cl /nologo /MD /DDLL dll00.c  /link /NOLOGO /DLL /OUT:PointCloud.dll
dll00.c
   Creating library PointCloud.lib and object PointCloud.exp

[prompt]>
[prompt]> dir /b
code00.py
code01.py
dll00.c
dll00.obj
PointCloud.dll
PointCloud.exp
PointCloud.lib

[prompt]>
[prompt]> :: CTypes
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ./code00.py
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] 064bit on win32

From C: Attempting to return 9724 points

Function <_FuncPtr object at 0x0000020CA1C40EE0> returned: 0

Points: 9724
Data (3 points):
  0: 0.0 1.0 2.0
  4862: 14586.0 14587.0 14588.0
  9723: 29169.0 29170.0 29171.0

Done.


[prompt]>
[prompt]> :: CFFI
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ./code01.py
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] 064bit on win32

From C: Attempting to return 9740 points

Function <cdata 'int(*)(void *, int)' 0x00007FFB33351020> returned: 0

Points: 9740
Data (3 points):
  0: 0.0 1.0 2.0
  4870: 14610.0 14611.0 14612.0
  9739: 29217.0 29218.0 29219.0

Done.

Might also want ro check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for a(nother) pitfall: this time when working with CTypes (calling functions).