How to cache Poetry virtual environments in GitLab CI/CD?

568 views Asked by At

I want to cache Poetry virtual environments between builds of my Python application. I have configured the pipeline so that the virtual environment is recreated every time pyproject.toml changes (poetry.lock is not comitted):

default:
  cache:
    - key:
        files:
        - lego/pyproject.toml
      paths:
        - lego/.venv
        - artifacts/build/layer/build/.venv
    - paths:
        - artifacts/pypoetry

The virtual environment is created as follows:

poetry config virtualenvs.in-project true
poetry install

This works locally, but it fails in GitLab CI/CD on the second pipeline run:

$ LOG_LEVEL=ERROR poetry run pytest tests -n auto -v --cov=./src --junitxml=lego-report.xml

ImportError while loading conftest '/builds/5565557/lddpro-bff/lego/tests/conftest.py'.
tests/conftest.py:22: in <module>
   from lego.common.aws import S3, EventBridge, SSM, Secrets
E ModuleNotFoundError: No module named 'lego'

The reason is that pytest cannot locate the root package (lego). However, I have verified that the root package is located in the virtual environment (.venv):

$ cat .venv/lib/python*/site-packages/lego.pth
/builds/5565557/lddpro-bff/lego/src

Here lego.pth includes the current job ID in the path.

How can I properly cache Poetry virtual environments between CI/CD pipeline runs?

Job configuration:

lego:
  extends: .unittest
  variables:
    REPORT_NAME: lego-report.xml
  script:
    - cd lego
    - poetry config virtualenvs.in-project true && poetry install
    - poetry env info
    - env LOG_LEVEL=ERROR poetry run pytest tests -n auto -v --cov=./src --junitxml=${REPORT_NAME}
    - mv -f ${REPORT_NAME} ${CI_PROJECT_DIR}/ || echo ""
    - cd ${CI_PROJECT_DIR}
    - mkdir -p artifacts
    - cd tools
    - . ./build_deploy.sh --command=build --debug
  artifacts:
    paths:
      - artifacts/lego-function.zip
      - artifacts/lego-layer.zip
    reports:
      junit: ${REPORT_NAME}
    expire_in: 7 days
  rules:
    - if: $CI_COMMIT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
0

There are 0 answers