Multiple pipelines for the same branch

Question

GitLab CI offers very nice flexibility and multiple features. Because of that is not obvious how particular configurations should be achieved using GitLabCI yaml, scheduler, etc.

I am interested to setup a workflow, which I am presenting below in a simplified way.

I use only master branch, where I want my primary pipeline to kick in after every commit. This pipeline will build and test my package. I want to also have a scheduled nightly pipeline which publish website of my package. So it takes the most recent artifacts, generate documentation and publish that together to GitLab pages.

So, basically I want to have two different pipelines being run on the same branch. Primary is the regular one, and the secondary is the one run by scheduler.

In my real use case I want to have 2 scheduled nightly pipelines and one weekly pipeline, so including primary one (every commit), total 4 pipelines, still for the same branch.

Using only/rules doesn't really address that. There is a new workflow command, which could in theory, be applied here, by branching

if scheduled then
  include nightly.gitlab-ci.yml
else
  include primary.gitlab-ci.yml

But I don't think it is possible.

What could be the most user friendly solution is to give scheduled pipelines an option to choose custom .gitlab-ci.yml file rather than starting default one.

Any ideas how to achieve configuration I am looking for?

The only way I have in mind is to keep having another branch, clone of master, where I need to replace .gitlab-ci.yml every evening. Which requires extra workstation to automate that.

Answer 1

If you have pipeline which launches after every commit then just add deploy stage to it which will only be executed for scheduled triggers. That way you will need only one CI configuration:

build_job:
...
test_job:
...
deploy_job:
...
  only:
    - schedules

I understand your original question is if you can run deploy job on scheduled pipelines only but why not test it and build anew in such case? It is usually a good practice to test and build your artifacts just before deploy especially if it doesn't take too much time.

Answer 2

I want to have 2 scheduled nightly pipelines and one weekly pipeline, so including primary one (every commit), total 4 pipelines, still for the same branch

I understand you want 4 pipelines such as:

• Primary pipeline run for commited changes
• Nightly 1 pipeline running every night
• Nightly 2 pipeline running every night, different than Nightly 1
• Weekly pipeline running every week

For Primary you can leverage rules to only run jobs on commited change with $CI_PIPELINE_SOURCE == "push":

# Primary job running only on push (i.e. on pushed commits)
# It won't run on schedule
primary_pipeline_job:
  stage: build
  script:
  - echo "I will run only on commited changes"
  # [...]
  rules:
  - if $CI_PIPELINE_SOURCE == "push"

For multiple scheduled pipelines, you can leverage both $CI_PIPELINE_SOURCE == "schedule" and custom variables set on your schedule config such as:

# nightly_pipeline_1_job_A and job_B will be run on scheduled pipelines
# only on schedule and when RUN_NIGHTLY_PIPELINE_1 == "true"
nightly_pipeline_1_job_A:
  # [...]
  rules:
  - if: $CI_PIPELINE_SOURCE == "schedule" && $RUN_NIGHTLY_PIPELINE_1 == "true"

nightly_pipeline_1_job_B:
  # [...]
  rules:
  - if: $CI_PIPELINE_SOURCE == "schedule" && $RUN_NIGHTLY_PIPELINE_1 == "true"

# nightly_pipeline_2_job_A will be run on scheduled pipelines
# only on schedule and when RUN_NIGHTLY_PIPELINE_2 == "true"
nightly_pipeline_1_job_B:
  # [...]
  rules:
  - if: $CI_PIPELINE_SOURCE == "schedule" && $RUN_NIGHTLY_PIPELINE_2 == "true"

# weekly_pipeline_job will be run on scheduled pipelines
# only on schedule and when RUN_WEEKLY_PIPELINE == "true"
weekly_pipeline_job:
  # [...]
  rules:
  - if: $CI_PIPELINE_SOURCE == "schedule" && $RUN_WEEKLY_PIPELINE == "true"

In your GitlabCI settings CI/CD > Schedules > [Your schedule] > Variables you can related variables. For example, for Nightly pipeline 1:

---

If you have multiple jobs you want to run on the same pipeline, instead of duplicating rules:, you can leverage extends such as:

.nightly_pipeline_1:
  rules:
  - if: $CI_PIPELINE_SOURCE == "schedule" && $RUN_NIGHTLY_PIPELINE_1 == "true"

nightly_pipeline_1_job_A:
  extends: .nightly_pipeline_1
  # [...]

nightly_pipeline_1_job_B:
  extends: .nightly_pipeline_1
  # [...]