I'm having a bad time installing some modules with poetry

Question

First, a bit of context

I have an old project managed with poetry and I wanted to install new modules. I tried installing them with poetry add <mymodule>... but it failed, poetry update failed, everything failed... So, after a big headache time, as always with poetry, I decided to start from scratch.

What I did

I installed the latest python 3.12

sudo add-apt-repository ppa:deadsnakes/ppa -y
sudo apt install python3.12-full
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.12 1

Then, the latest poetry (1.6.1) and created a new project

poetry 1.6.1: curl -sSL https://install.python-poetry.org | python3 -
poetry env remove <my_old_env>
poetry init (I selected python 3.12)
poetry env use /usr/bin/python3.12
poetry add <all_my_needed_modules>

And it failed again....

So I tried adding modules one by one and, at the end, I had 2 failing modules : pyarrow and pytickersymbols.

For pytickersymbols, it was pyyaml who failed when downgrading from 6.0.1 to 6.0

For pyarrow, poetry said it was a problem with wheel (and, as a matter of fact, it said the same thing with pyyaml 6.0)

With both of them, I had a message at the end :

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyarrow (13.0.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "pyarrow (==13.0.0)"'

Same for pyyaml (pip wheel --use-pep517 "pyyaml (==6.0)")

Really?

Well, I tried these 2 commands, pip wheel --use-pep517 "pyarrow (==13.0.0) and pip wheel --use-pep517 "pyyaml (==6.0)"... and it downloaded the .whl files just fine. So, after a bit of research, I found that I could force these files in the pyproject.toml configuration file :

[tool.poetry.dependencies]
pyyaml = { file = "whl/PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl" }
pytickersymbols = "^1.13.0"
pyarrow = { file = "whl/pyarrow-13.0.0-cp310-cp310-manylinux_2_28_x86_64.whl" }

And they installed without complaining.

And now?

It seems to work for pyyaml/pytickersymbols, but pyarrow is not functionnal : ImportError: Missing optional dependency 'pyarrow'. pyarrow is required for parquet

I don't know what to do from here...How can I install pyarrow ?

[EDIT] I guess I don't understand the installation mechanisms, but it seems poetry do not download wheels, but try to compile modules instead :

$ poetry add pyarrow --python=3.12

Using version ^13.0.0 for pyarrow

Updating dependencies
Resolving dependencies... (0.5s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing pyarrow (13.0.0): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  <string>:34: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-312
  creating build/lib.linux-x86_64-cpython-312/pyarrow
  copying pyarrow/filesystem.py -> build/lib.linux-x86_64-cpython-312/pyarrow
  copying pyarrow/pandas_compat.py -> build/lib.linux-x86_64-cpython-312/pyarrow

(...)

 copying pyarrow/src/arrow/python/visibility.h -> build/lib.linux-x86_64-cpython-312/pyarrow/src/arrow/python
  running build_ext
  creating /tmp/tmpbsvpo2s2/pyarrow-13.0.0/build/temp.linux-x86_64-cpython-312
  -- Running cmake for PyArrow
  cmake -DCMAKE_INSTALL_PREFIX=/tmp/tmpbsvpo2s2/pyarrow-13.0.0/build/lib.linux-x86_64-cpython-312/pyarrow -DPYTHON_EXECUTABLE=/tmp/tmpbfzby4k_/.venv/bin/python -DPython3_EXECUTABLE=/tmp/tmpbfzby4k_/.venv/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /tmp/tmpbsvpo2s2/pyarrow-13.0.0
  -- The C compiler identification is GNU 11.4.0
  -- The CXX compiler identification is GNU 11.4.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- System processor: x86_64
  -- Performing Test CXX_SUPPORTS_SSE4_2
  -- Performing Test CXX_SUPPORTS_SSE4_2 - Success
  -- Performing Test CXX_SUPPORTS_AVX2
  -- Performing Test CXX_SUPPORTS_AVX2 - Success
  -- Performing Test CXX_SUPPORTS_AVX512
  -- Performing Test CXX_SUPPORTS_AVX512 - Success
  -- Arrow build warning level: PRODUCTION
  -- Using ld linker
  -- Build Type: RELEASE
  -- CMAKE_C_FLAGS:  -Wall -fno-semantic-interposition -msse4.2  -fdiagnostics-color=always  -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
  -- CMAKE_CXX_FLAGS:  -Wno-noexcept-type  -Wall -fno-semantic-interposition -msse4.2  -fdiagnostics-color=always  -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized
  -- Generator: Unix Makefiles
  -- Build output directory: /tmp/tmpbsvpo2s2/pyarrow-13.0.0/build/temp.linux-x86_64-cpython-312/release
  CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
    Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Development.Module
    NumPy) (found version "3.12.0")
  Call Stack (most recent call first):
    /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
    /usr/share/cmake-3.22/Modules/FindPython/Support.cmake:3180 (find_package_handle_standard_args)
    /usr/share/cmake-3.22/Modules/FindPython3.cmake:490 (include)
    cmake_modules/FindPython3Alt.cmake:51 (find_package)
    CMakeLists.txt:255 (find_package)
  
  
  -- Configuring incomplete, errors occurred!
  See also "/tmp/tmpbsvpo2s2/pyarrow-13.0.0/build/temp.linux-x86_64-cpython-312/CMakeFiles/CMakeOutput.log".
  error: command '/usr/bin/cmake' failed with exit code 1
  

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:147 in _prepare
      143│ 
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│ 
      146│             if error is not None:
    → 147│                 raise error from None
      148│ 
      149│             return path
      150│ 
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyarrow (13.0.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "pyarrow (==13.0.0)"'.

and why "python 3.10" in at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py?