How to nest c-api extensions in a Python module and use them in that module?

Question

I want to create a Python 3 package called "mypack" that can be installed with pip. It has some c-extensions and some python code that calls those extensions. Here is my directory structure:

setup.py
mypack/__init__.py
mypack/mypack.py
foopkg/foo.cpp

The setup.py file has the following code:

from setuptools import setup, Extension

PACKAGE_NAME = 'mypack'

module = Extension('foo',
                language = "c++",
                sources = ['foopkg/foo.cpp'])

setup(name=PACKAGE_NAME,
      version='1.0',
      packages=[PACKAGE_NAME],
      ext_package=PACKAGE_NAME,
      ext_modules=[module],
      include_package_data=True)

I adapted this code from another related question where the user wanted to import the extensions using something like mypack.xxx, as I do. In the c-api extension, I have successfully compiled it and made it work as a stand-alone extension module, but I am having trouble incorporating it into a larger package. It defines two functions make_array and print_array. For brevity I removed the function code and just included the stuff that Python needs:

...

static PyMethodDef FooMethods[] = {
    { "make_array", make_array, METH_VARARGS, "Put number in array"},
    { "print_array", print_array, METH_VARARGS, "Print number from array"},
    { NULL, NULL, 0, NULL}
};

static struct PyModuleDef foomodule = {
    PyModuleDef_HEAD_INIT,
    "foo",
    "Make/Print array",
    -1,
    FooMethods
};

PyMODINIT_FUNC PyInit_foo(void)
{
    return PyModule_Create(&foomodule);
}

I want to be able to import this extension within the package to use it (this is mypack.py):

import mypack.make_array
import mypack.print_array

def dostuff():
    array = make_array(10)
    print_array(array)

Lastly, my __init__.py file contains from .mypack import dostuff.

However, when I install with pip, and try to run a test script that imports mypack it complains about the imports in mypack.py, whether I use foo.xx or mypack.xx. I have built other packages with a nested structure that use code from other python files in the module using __init__.py and using imports. But I am a bit baffled about how to do this with c-extensions.

Answer 1

The fact that the module is compiled from C makes no difference to how you write the import statement. The line...

import mypack.make_array

tries to import a module named "make_array" found in a package named "mypack". That's not what you have. You have a module named "foo" in a package named "mypack", and two objects (methods) in that module named "make_array" and "print_array."

It's much simpler if you remove everything from __init__.py and make it work that way. You can add stuff to the init file later if you want to improve the syntax of your import statements. You want...

from mypack.foo import make_array, print_array

or

import mypack.foo as MYPACK

In the first case the methods are now in the global namespace; in the second case you access them as MYPACK.make_array and MYPACK.print_array.

Also I see a problem with naming your C-extension file mypack and also naming your python file mypack.py. There can be only one module with that name in a package. One solution is to prefix an underscore to the C file name. Then in mypack.py you write, for example...

from ._mypack import make_array

This puts make_array into the namespace of myarray, and client code doesn't have to know, or care, that it's written in C.