Working with multiple Python versions **including development header files**?

Question

I develop Python application with support for different Python versions. I use pyenv to have multiple Python versions on my system (Linux, if that matters) and to switch between them.

Now I also need to build Python packages with extensions, as not all of them yet publish wheels for the recent Python version. And my system has development headers which is guaranteed to work with only one Python version (package python3-dev, which, on my system, serves Python 3.10.6). So attempts to build (some) packages in e.g. Python 3.12 pyenv environment fail.

Ideally, I want to be able to say:

$ pyenv use 3.12
$ pip wheel <package>  # <-- and GCC is using headers for Python 3.12, not for the version for which the development headers are installed system-wide

Is this a reasonable expectation? How can it be achieved?

Answer 1

One option which certainly satisfies this (whether it fits your workflow is a separate question) is to use docker containers.

For python, this is relatively easy:

docker run -it python:3.10 bash

Drops you into a bash shell where python is python 3.10, gcc is installed, and the python headers are present (try python-config --includes to check).

To keep your changes, you can commit the container, or you can move the 'setup' to a single dockerfile and parametrize the base:

#syntax=docker/dockerfile:1.4

ARG version=3.10
FROM python:${version}
RUN install-stuff

Now you can build multiple images by supplying $version at build time:

docker build --build-arg version=3.12 -t dev:3.12

Docker will pull down the relevant base container and build your image on top of it.

This is possible and easy in this case, because python provides a set of high quality docker images for every version you might care about.