Vendoring Python itself in-repos, cross-platform

20 views Asked by At

The majority of our code base is in C++, we rely increasingly heavily on Python tho for our automation, CI, and pipelines. We operate a large mono-repos that is almost entirely run-from-repos - we don't have an installer; our users (employees) run various MacOS, Windows and Ubuntu environments; and run from repos means they go backwards, forwards, and sideways in time as well as having different product/feature branches checked out side-by-side simultaneously. We also have several 1st and 3rd party tools that embed Python into C++ apps.

At this point in our snake-herding adventures, I believe we need to vendor the specific Python version the repository is intended to operate against at that specific version. Increasingly, of late, Python seems to really need to be installed though.

We have - of course - tried pip, conda, mamba, pyenv, virtualenv, self-managing solutions, but the key word here is "pipeline": a lot of _sub_process happens and there, when we've almost gotten things to work, all bets terminate. We had a nearly-working virtualenv solution, but literally the first 3rd party package we depended on - psutil - quickly lead to an in-repos wheel stock that started to dwarf even our imagemagik libs.

What we want is to be able to ensure our python code gets executed by one particular python interpreter with one particular set of paths and packages anchored to the path of the python code itself (argv[0] / file is about all we can rely on), without having to provision and maintain a massive number of variants of the "current" Python build into the repos at all times.

IS it feasible to be vendoring, say, Python 3.10 as a green-install across such a relatively broad variety of targets, and if so what tooling can we use to buy down the management costs, and avoid huge per-script startup overheads of trying to return the environment to expectations?

0

There are 0 answers