I am getting this error when running any poetry executables
Traceback (most recent call last):
File "/root/.local/bin/poetry", line 5, in <module>
from poetry.console.application import main
File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py", line 11, in <module>
from cleo.application import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'
My container is built using this logic.
FROM gcr.io/dataflow-templates-base/python38-template-launcher-base:flex_templates_base_image_release_20230508_RC00
ARG DIR=/dataflow/template
ARG dataflow_file_path
ARG PROJECT_ID
# environment to pull the right containers
ARG ENV
ARG TOKEN
ENV COMPOSER_$ENV=1
# copying over necessary files
RUN mkdir -p ${DIR}
WORKDIR ${DIR}
COPY transform/dataflow/${dataflow_file_path}.py beam.py
COPY deploy/dataflow/poetry.lock .
COPY deploy/dataflow/pyproject.toml .
# env var in order to use custom lib, for more info, see:
# https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#set_required_dockerfile_environment_variables
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${DIR}/beam.py"
ENV FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGES=""
ENV FLEX_TEMPLATE_PYTHON_PY_OPTIONS=""
ENV PIP_NO_DEPS=True
# install poetry
RUN curl -sSL https://install.python-poetry.org | python -
ENV PATH "/root/.local/bin/:${PATH}"
RUN poetry --version
I have tried uninstalling it and the suggestions from:
- https://medium.com/@life-is-short-so-enjoy-it/python-poetry-modulenotfounderror-no-module-named-cleo-1f924239b06c
- I am getting no module name cleo when trying to do "poetry add"
- https://github.com/python-poetry/poetry/issues/3071
- https://github.com/python-poetry/poetry/issues/2991
These aren't really applicable because they are poetry executables in a non-docker environment but not really sure what else to do . I have an Dataflow SDK that was build from apache/beam_python3.8_sdk:2.45.0 that has the same logic and it is working.
I disregarded the last RUN command and built the container, these are the outputs of some checks
❯ docker run --rm --entrypoint /bin/bash dataflow -c 'which poetry'
/root/.local/bin/poetry
❯ docker run --rm --entrypoint /bin/bash dataflow -c 'poetry'
Traceback (most recent call last):
File "/root/.local/bin/poetry", line 5, in <module>
from poetry.console.application import main
File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py", line 11, in <module>
from cleo.application import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'
❯ docker run --rm --entrypoint /bin/bash dataflow -c 'which python'
/usr/local/bin/python
My assumption is that the poetry executables are importing in a virtualenv that the other libraries aren't installed to.
UPDATE:
I went down a rabbit hole and did this
RUN pip install --no-cache-dir poetry cleo rapidfuzz importlib_metadata zipp crashtest
And running poetry --version worked but poetry config virtualenvs.create false or any other command will throw this error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/poetry", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 409, in main
exit_code: int = Application().run()
File "/usr/local/lib/python3.8/site-packages/cleo/application.py", line 338, in run
self.render_error(e, io)
File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 180, in render_error
self.set_solution_provider_repository(self._get_solution_provider_repository())
File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 398, in _get_solution_provider_repository
from poetry.mixology.solutions.providers.python_requirement_solution_provider import ( # noqa: E501
File "/usr/local/lib/python3.8/site-packages/poetry/mixology/__init__.py", line 5, in <module>
from poetry.mixology.version_solver import VersionSolver
File "/usr/local/lib/python3.8/site-packages/poetry/mixology/version_solver.py", line 8, in <module>
from poetry.core.packages.dependency import Dependency
ModuleNotFoundError: No module named 'poetry.core'