Using Python setuptools to put Cython-compiled pyd files in their original folders?

Question

I'm trying to build the setup.py package for my Python 3.4 app which contains a Cython module. My plan is to have a setup.py file which requires Cython and compiles the .pyx file with the idea that typically I would just run that on Win32 (x86 & x64) and Mac and then generate platform wheels and upload them to PyPI, so regular users wouldn't need Cython or to compile anything.

Currently I can get the setup.py script to build the .pyd file, but it doesn't put the built .pyd file in the same location as the original .pyx file which is where I need it to be.

I tried manually copying the .pyd file, but then when I generate the wheel via bdist_wheel I get a pure Python wheel which I think is wrong since there are different version of my built .pyx file depending on the platform. I also tried subclassing the Distribution class and forcing is_pure() to return False (as described here in the Building Wheels section), but that doesn't work. (Still generates a pure Python wheel.)

I assume that if I can get the setup.py script to put the compiled file in the right location, then the wheel will not be pure Python and everything will be fine.

So my questions are:

1. Is there some kind of setting in setup.py to tell it to put the compiled Cython files into the same locations as the source files?
2. Or is there a way to copy the compiled files to the location I need them but for the wheel to still be built as non-pure?
3. Or am I going about this in the wrong way. Should I change my Python code to pull the compiled packages from somewhere else? (That doesn't seem right to me.)
4. [EDIT] Or should I put the compiled binary .pyd / .so / .dylib into the package, add logic to get the right .pyd based on architecture at runtime, and then have a pure Python wheel? (That also doesn't seem right to me.)

More information on my setup: I'm using setuptools (not distutils) in my setup.py script. A snippet of my setup.py is here (with a lot of stuff removed to keep it to the stuff that's relevant):

from setuptools import setup, Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext

...

extensions = [
    Extension(name='audio_interface',
              sources=['mpf/mc/core/audio/audio_interface.pyx'],
              include_dirs=include_dirs,
              library_dirs=library_dirs,
              libraries=libraries,
              extra_objects=extra_objects,
              extra_compile_args=extra_compile_args,
              extra_link_args=extra_link_args),

...

setup(
      ...
      ext_modules=cythonize(extensions),
      cmdclass= {'build_ext': build_ext},
      ...

Thanks! Brian

Answer 1

I found a workaround which solves the immediate problem which is to add --plat-name to the python setup.py bdist_wheel command. (This was added in wheel 0.27.0. No idea why it didn't come up in all my searches before.)

So this solves my problem via Item #2 on my list. I compile the pyx to .pyd or .so, copy it back to the proper location, and then use --plat-name to create a platform-specific wheel.

However my question still somewhat remains because if we add more Cython modules, do we compile and then just copy them all back to their original locations manually, or is there a way for them to be compiled and go automatically to their original locations? Also the way I'm doing it involves two different setup.py files--one to compile and a second to install the package.

So I'm adding this here for the benefit of future searchers but my original questions still remain.

Answer 2

I had a similar issue, and I have found a solution that could work also for you.

From the official Python documentation for distutils.core.Extension, the name argument is:

the full name of the extension, including any packages — ie. not a filename or pathname, but Python dotted name

So, if you modify your setup.py to:

...

extensions = [
    Extension(name='mpf.mc.core.audioaudio_interface',  # using dots!
              sources=['mpf/mc/core/audio/audio_interface.pyx'],
              include_dirs=include_dirs,
              library_dirs=library_dirs,
              libraries=libraries,
              extra_objects=extra_objects,
              extra_compile_args=extra_compile_args,
              extra_link_args=extra_link_args),

...

you will have the compiled binary in the location that you want.