Linked Questions

Popular Questions

Unable to install poetry in dataflow

Asked by At

I am getting this error when running any poetry executables

Traceback (most recent call last):
  File "/root/.local/bin/poetry", line 5, in <module>
    from poetry.console.application import main
  File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py", line 11, in <module>
    from cleo.application import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'

My container is built using this logic.

FROM gcr.io/dataflow-templates-base/python38-template-launcher-base:flex_templates_base_image_release_20230508_RC00
ARG DIR=/dataflow/template
ARG dataflow_file_path
ARG PROJECT_ID
# environment to pull the right containers
ARG ENV
ARG TOKEN
ENV COMPOSER_$ENV=1

# copying over necessary files
RUN mkdir -p ${DIR}
WORKDIR ${DIR}
COPY transform/dataflow/${dataflow_file_path}.py beam.py
COPY deploy/dataflow/poetry.lock .
COPY deploy/dataflow/pyproject.toml .

# env var in order to use custom lib, for more info, see:
# https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#set_required_dockerfile_environment_variables
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${DIR}/beam.py"
ENV FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGES=""
ENV FLEX_TEMPLATE_PYTHON_PY_OPTIONS=""
ENV PIP_NO_DEPS=True

# install poetry
RUN curl -sSL https://install.python-poetry.org | python -
ENV PATH "/root/.local/bin/:${PATH}"
RUN poetry --version

I have tried uninstalling it and the suggestions from:

These aren't really applicable because they are poetry executables in a non-docker environment but not really sure what else to do . I have an Dataflow SDK that was build from apache/beam_python3.8_sdk:2.45.0 that has the same logic and it is working.

I disregarded the last RUN command and built the container, these are the outputs of some checks

❯ docker run --rm --entrypoint /bin/bash dataflow -c 'which poetry'
/root/.local/bin/poetry
❯ docker run --rm --entrypoint /bin/bash dataflow -c 'poetry'
Traceback (most recent call last):
  File "/root/.local/bin/poetry", line 5, in <module>
    from poetry.console.application import main
  File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py", line 11, in <module>
    from cleo.application import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'
❯ docker run --rm --entrypoint /bin/bash dataflow -c 'which python'
/usr/local/bin/python

My assumption is that the poetry executables are importing in a virtualenv that the other libraries aren't installed to.

UPDATE:

I went down a rabbit hole and did this

RUN pip install --no-cache-dir poetry cleo rapidfuzz importlib_metadata zipp crashtest

And running poetry --version worked but poetry config virtualenvs.create false or any other command will throw this error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/poetry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 409, in main
    exit_code: int = Application().run()
  File "/usr/local/lib/python3.8/site-packages/cleo/application.py", line 338, in run
    self.render_error(e, io)
  File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 180, in render_error
    self.set_solution_provider_repository(self._get_solution_provider_repository())
  File "/usr/local/lib/python3.8/site-packages/poetry/console/application.py", line 398, in _get_solution_provider_repository
    from poetry.mixology.solutions.providers.python_requirement_solution_provider import (  # noqa: E501
  File "/usr/local/lib/python3.8/site-packages/poetry/mixology/__init__.py", line 5, in <module>
    from poetry.mixology.version_solver import VersionSolver
  File "/usr/local/lib/python3.8/site-packages/poetry/mixology/version_solver.py", line 8, in <module>
    from poetry.core.packages.dependency import Dependency
ModuleNotFoundError: No module named 'poetry.core'

Related Questions