Dependencies and local package not available when building notebooks in GitHub actions workflow

Question

I am getting back to Python but new to poetry and papermill.

We have repository hosted on GitHub which is a Python package with pyproject.toml which is private, once public and if this issue persists, will provide URL. We also have some notebooks in the same repo.

The steps are simply:

# installs package successfully but in an env
poetry install -E all
# run papermill under poetry 
poetry run papermill SomeNotebook.ipynb output_folder/file.txt
# same issues for
papermill SomeNotebook.ipynb output_folder/file.txt

When trying to build the notebooks using papermill I cannot:

1. Get the package dependencies to be available during the builds
2. Get the local package itself to be available during the builds

As requested in comment, the error is simply:

# for deps
ModuleNotFoundError                       
Traceback (most recent call last)
Cell In[1], line 1
----> 1 import pandas as pd

and exactly the same for the local package:

ModuleNotFoundError                       
Traceback (most recent call last)
Cell In[1], line 1
----> 1 from mypackage.plots import *

The poetry install -E all in the workflow apprently installs the package itself and its dependencies in an env and requires poetry shell. But doing poetry shell is not possible as I get Inappropriate ioctl for device. I have tried activating the env via source $(poetry env info --path)/bin/activate, and tried poetry config virtualenvs.create false I get the same errors.

Can you help me build the notebooks with both deps and local/current package available during papermill step?

Thank you

Answer 1

This is not the answer but a solution. What we decided to use is to avoid poetry and use pip. I will leave the question there for in case the right answer comes.

So the above code is now:

# installs deps & package
pip install -e '.[all]'
# run papermill directly
papermill SomeNotebook.ipynb output_folder/file.txt
# passed github actions