How to use gitlab-ci to manage test/construction of interdependent wheels

Question

I have 3 python packages proj1, proj12 and proj13. Both proj12 and proj13 depend on proj1 (with from proj1.xxx import yyy).

The 3 projects are on a private gitlab instance, each one has it's own .gitlab-ci.

In proj1 http://gitlab.me.com/group/proj1/.gitlab-ci.yml we run unittest and create a wheel exposed as an artifact::

# http://gitlab.me.com/group/proj1/.gitlab-ci.yml
image: python:2
mytest:
  artifacts:
    paths:
    - dist
  script:
  - apt-get update -qy; apt-get install -y python-dev python-pip
  - pip install -r requirements.txt
  - python setup.py test
  - python setup.py bdist_wheel
look:
  stage: deploy
  script:
  - ls -lah dist

For proj12 and proj13 in e.g. http://gitlab.me.com/group/proj12/.gitlab-ci.yml we would like to run tests too, but I need to install proj1 wheel to make it run.

All 3 projects are in the same gitlab private group.

What is the gitlab way to do this ?

• to pass the proj1 wheel to the proj12 with an artifact
  • in this case I don't know how to call/get the artifact in http://gitlab.me.com/group/proj12/.gitlab-ci.yml ? It's the same gitlab, the same group, but a different project.
• Use a gitlab Secret Variable to store ssh_keys to clone proj2 in proj12/.gitlab-ci.yml ?
  • related to https://gitlab.com/gitlab-org/gitlab-ce/issues/4194
  • this does not take benefit of the fact that proj1, proj12 and proj13 are in the same gitlab and same group, the person who do the build for one project as credentials to do the others. All 3 are connected by the user private token.

I try to avoid to have to deploy devpi or pypiserver like solutions.

So I'm looking on what to write in the proj12 .gitlab-ci.yml to get the dist/proj1-0.42-py2-none-any.whl wheel from the proj1 precedent build::

# http://gitlab.me.com/group/proj12/.gitlab-ci.yml
image: python:2
mytest12:
  script:
  - apt-get update -qy; apt-get install -y python-dev python-pip
  - pip install -r requirements.txt
  - pip install .
  - => some way here to get the proj1 wheel 
  - pip install proj1-0.42-py2-none-any.whl
  - python setup.py test

Links related to our issue:

• Allow access to build artifacts by using restricted access tokens https://gitlab.com/gitlab-org/gitlab-ce/issues/19628
• "People need to be able to share links to artifacts based on a git ref (branch, tag, etc.), without knowing a specific build ID https://gitlab.com/gitlab-org/gitlab-ce/issues/4255
• https://docs.gitlab.com/ce/api/ci/builds.html#upload-artifacts-to-build
• download-the-artifacts-file https://docs.gitlab.com/ce/api/builds.html#download-the-artifacts-file https://gitlab.com/gitlab-org/gitlab-ce/issues/22957

Answer 1

You have two ways you can do it:

• Pass the object from previous build using the artifacts (works inside the same project only)
• Build a docker image with your packages pre-installed in a git job, store it in the in-built registry and use that to run build in your other projects.
• Clone the repository

I would advise passing as an artifact since then you will have it build exactly in the pipeline you are running. As for the cloning, AFAIK you don't need any workaround when cloning submodules but for cloning other repositories I would go with ssh deploy key as it's connected with a repo and not a user like the private token.