How to package a nested Python module using scikit-build?

Question

scikit-build is failing to package a python project for me. Here's the layout

hello-cpp
├── src
|   └── hello
|       ├── OtherModule
|       |   └── __init__.py  # empty
|       └── __init__.py      # empty
|       └── hello.cpp
├── CMakeLists.txt
└── setup.py

contents of setup.py:

from skbuild import setup
from setuptools import find_packages

setup(
    name="hello-cpp",
    version='0.0.1',
    description="a minimal example package (cpp version)",
    author="Me",
    license="MIT",
    packages=find_packages('src'),
    package_dir={'': 'src'},
    python_requires=">=3.7",
    zip_safe=False,
)

content of CMakeLists.txt:

cmake_minimum_required(VERSION 3.4...3.22)

project(hello)

find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(example src/hello/hello.cpp)
install(TARGETS example LIBRARY DESTINATION src/hello)

contents of src/hello/hello.cpp:

#include <pybind11/pybind11.h>

int add(int i, int j) {
    return i + j;
}

PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin"; // optional module docstring

    m.def("add", &add, "A function that adds two numbers");
}

When running python -c "from hello import example" I get ModuleNotFoundError: No module named 'hello'.

Removing src/hello/OtherModule/__init__.py allows scikit-build to package the module, but I want to have hello.OtherModule, so this is not an option.

I have tried installing this via pip install -e . as well as python setup.py develop and python -m pip install --editable . inside a conda environment built with conda create --name so_example python=3.10 scikit-build ninja and nothing else installed. The hello.hello C++ module appears to be properly built by pybind11, but the hello package itself is not properly installed, despite pip claiming that it

Successfully installed hello-cpp-0.0.1

Answer 1

FYI, I copied your files into a directory and added a pyproject.toml file:

[build-system]
requires = ["setuptools", "scikit-build", "pybind11"]
build-backend = "setuptools.build_meta"

And tried this:

$ virtualenv .venv
$ .venv/bin/pip install .
$ tree .venv/lib/python3.11/site-packages/hello
.venv/lib/python3.11/site-packages/hello
├── OtherModule
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-311.pyc
├── __init__.py
├── __pycache__
│   └── __init__.cpython-311.pyc
└── example.cpython-311-darwin.so

And it seems fine. Note that setuptools + manual CMake addition, scikit-build, and scikit-build-core all do not support editable installs (-e/--editable) since Setuptools 64. If you force the old implementation, it might sort-of work, but really isn't ideal still - you aren't getting much "editable" out of it. Scikit-build-core will have a proper editable installation mechanism soon (within a month or two).