std::bad_alloc after replacing boost:python function wrapper with Python/C API

Question

I had a function in C which I used to extend python, previously using the BOOST_MODULE function to accomplish this. This error came up when transitioning to the python-C API. I am certain that the run_mymodule function runs fine without this wrapper.

static PyObject * wrap_run_mymodule(PyObject *, PyObject *args) {
    char *file1, *file2, *file3;
    PyObject *tmpp;
    if(!PyArg_ParseTuple(args, "sssO", &file1, &file2, &file3, &tmpp))
        return NULL;
    return Py_BuildValue("i", run_mymodule(file1, file2, file3, tmpp));
}

static PyMethodDef myModule_methods[] = {
    {"run_mymodule", (PyCFunction) wrap_run_mymodule, METH_VARARGS},
    {NULL, NULL}
};

extern "C" void initmymodule(void)
{
    (void) Py_InitModule("mymodule", myModule_methods);
}

the declaration of the function is of this form: int run_mymodule(char *file1, char *file2, char *file3, PyObject *tmpp)

Here is the exact error message I get:

 python(35137,0x7fff76453310) malloc: *** error for object 0x10afcfb78: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
 Abort trap: 6

How can I solve this problem? Where is this malloc error coming from? In python, I am passing strings as the first three arguments, and a python class as the fourth argument. Of course, I am happy to put probes into my code.

SanderMertens suggested I post the valgrind output-

$ valgrind python test_mymodule.py
==30715== Memcheck, a memory error detector
==30715== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==30715== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==30715== Command: python test_mymodule.py
==30715== 
==30715== Syscall param posix_spawn(pid) points to unaddressable byte(s)
==30715==    at 0x3D266E: __posix_spawn (in /usr/lib/system/libsystem_kernel.dylib)
==30715==    by 0x100001DC2: ??? (in /usr/local/bin/python)
==30715==    by 0x25E5FC: start (in /usr/lib/system/libdyld.dylib)
==30715==    by 0x1: ???
==30715==    by 0x1000138CF: ???
==30715==    by 0x104803AD1: ???
==30715==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==30715== 
Python(30715,0x7fff74a8e310) malloc: *** mach_vm_map(size=140735173898240) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
 Abort trap: 6

Answer 1

As has been explained in several of the comments now, you have a memory corruption issue. Key indicators are:

• The problem reproduces erratically.
• Exact symptoms change from unexpectedly bad values being passed into some functions, through to bizarrely large memory allocation failures.
• These symptoms don't reproduce for others when run_module has a trivial implementation.

Given that valgrind isn't pinpointing it for you, it's probably a stack corruption. You could look for places where your code is calling the failed function (from valgrind) and look at what your code did before that, but this will be slow.

Your best bet at this stage is now to use the stack validation tools out there. For example, assuming you're using gcc, try the address sanitizer or mudflap features (depending on which version you're using).