A package which calls a C function from a subdirectory

Question

I have a problem building a package which contains a subpackage with a C function. I've built a test case to illustrate it.

The directory tree is:

README.rst
MANIFEST.in
setup.py
testcase/
  get_triple.py
  __init__.py
  spin/
    __init__.py
    setup.py
    spin.c
    spin.h

Here are the contents of the files. First in the testcase directory.

get_triple.py:

import spin
print "here is the spin dir"
print dir(spin)

def get_triple(number):

    """ triple that number with a C func """

    new_number = spin.make_that_triple(number)

    return new_number

__init__.py:

from .get_triple import *

setup.py:

from setuptools import setup

def readme():
    with open('README.rst') as f:
        return f.read()

setup(name='testcase',
    version='0.1',
    description='test the C function called from a package',
    classifiers=[
        'Development Status :: 0 - Alpha',
        'License :: Friand_Dur',
        'Programming Language :: Python :: 2.7',
        'Topic :: Scientific/Engineering :: Bio-Informatics',
    ],
    author='test',
    author_email='test',
    license='test',
    package_dir = {
        'testcase': 'testcase',
        'testcase.spin': 'testcase/spin',
    },
    packages=['testcase', 'testcase.spin'],
    install_requires=[
        # nothing, it's a test
    ],
    zip_safe=False)

And from the spin directory.

spin.c:

#include <stdio.h>
#include <stdlib.h>
#include <python2.7/Python.h>
#include <math.h>
#define PY_ARRAY_UNIQUE_SYMBOL get_spinangle_traj_ARRAY_API
/* This one makes the compilation crash if not commented, despite the warning message if it is */
//~ #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include "numpy/arrayobject.h"
/* Not typecasting anything */
#include "spin.h"

/* Docstring */
static char module_docstring[] =
    "triple using C";
static char make_that_triple_docstring[] =
    "triple using C";

static PyObject *make_that_triple(PyObject *self, PyObject *args){

    /* Just get a number */
    int number;
    double tripled_number = 0.0;

    /* Should get the python objects */
    if (!PyArg_ParseTuple(args,"i",&number)){
        PyErr_SetString(PyExc_ValueError,"Error while parsing the trajectory coordinates in get_spinangle_traj");
        return NULL;
    }

    /* Call the external C function to get the triple */
    tripled_number = tripleeee(number);

    PyObject *ret = Py_BuildValue("d", tripled_number);
    return ret;
}

double tripleeee(int number){

    return number * 3;

}

static PyMethodDef module_methods[] = {
    {"make_that_triple",make_that_triple,METH_VARARGS,make_that_triple_docstring},
    {NULL,NULL,0,NULL}
};

PyMODINIT_FUNC initspin(void)
{
    PyObject *m = Py_InitModule3("spin",module_methods,module_docstring);
    if (m == NULL)
        return;
    /* Load Numpy */
    import_array();
}

spin.h:

static PyObject *make_that_triple(PyObject *self, PyObject *args);
double tripleeee(int number);

setup.py:

from distutils.core import setup, Extension
import numpy.distutils.misc_util

setup(
    ext_modules=[Extension("spin", ["spin.c"])],
    include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs(),
)

And the __init__.py is empty.

I first compiled the C package using python setup.py build_ext --inplace in the spin subdirectory and then the main package using sudo python setup.py install.

I also have a test script:

#!/usr/bin/python
# -*- coding: utf-8 -*-
import testcase
print "here is the testcase dir"
print dir(testcase)

# Let's multiply this by 3
number = 25

new_number = testcase.get_triple(number)

print new_number

Which ... doesn't work:

here is the spin dir
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
here is the testcase dir
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'get_triple', 'spin']
Traceback (most recent call last):
  File "./spindihed_totable_pl60.py", line 17, in <module>
    new_number = testcase.get_triple(number)
  File "/usr/local/lib/python2.7/dist-packages/testcase-0.1-py2.7.egg/testcase/get_triple.py", line 9, in get_triple
    new_number = spin.make_that_triple(number)
AttributeError: 'module' object has no attribute 'make_that_triple'

It can't access the C functions within spin from the testcase package even though the spin within it does exist, and I've been stuck at this step for a while. When I cd into the spin subdirectory, I can however use the C function, which seems to tell me it's not a C problem:

>>> import spin
>>> print spin.make_that_triple(3)
9.0

What did I miss ? I feel that there is something very basic that I missed. I also put everything on git to make the test easier: https://github.com/msidore/testcase

Thanks for your help

Answer 1

You've got a few issues here. After building the --inplace, you have created an extension module in testcase/spin/spin.so, so the corresponding module is testcase.spin.spin:

>>> import testcase.spin.spin
>>> dir(testcase.spin.spin)
['__doc__', '__file__', '__name__', '__package__', 'make_that_triple']

But since your top-level setup.py doesn't know about the extension module, it won't get installed if you python setup.py install. I suggest re-arranging things like this:

setup.py
testcase/get_triple.py
testcase/__init__.py
testcase/spin.c
testcase/spin.h

And modify your top-level setup.py so that it knows about the extension module:

from setuptools import setup, Extension
import numpy.distutils.misc_util

setup(name='testcase',
      packages=['testcase'],
      ext_modules=[Extension("testcase.spin", ["testcase/spin.c"])],
      include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs(),
      zip_safe=False)

With these changes in place, I can:

virtualenv .venv
. .venv/bin/activate
pip install numpy
pip install .
python testscript.py

And I get as output from testscript.py:

here is the spin dir
['__doc__', '__file__', '__name__', '__package__', 'make_that_triple']
here is the testcase dir
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'get_triple', 'spin']
75.0