Concourse CI: project structure

Question

Scenario:

I have been working on implementing a concourse ci pipeline for over a month now and my single yml file has grown quiet a bit. I understand that it is best practice to breakup the pipeline into several files and reference them in your pipeline.

Question:

Can someone please provide what the best practice is to structuring your concourse ci pipeline?

My thought process:

offering-pipeline | |_ ci: | | | |_ images: | | |_ Dockerfile | | | |_ misc: | | |_ python-requirements.txt | | | |_ ci-pipeline.yml | |_ project: |_ project-pipeline.yml | |_ jobs | |_ scripts : | |_ build: | |_ build_xyz. | |_ deploy: | |_ deploy_xyz. | |_ test: | |_ test_xyz. | |_ publish: |_ publish_xyz.

Thanks,

-Abe.

Answer 1

A First step would be to extract all tasks to files. I have a tasks folder, a templates folder, and a script folder for each pipeline. Above these (in the root) I have a pipeline.yml containing the pipeline root structure, and a Makefile and Makefile.constants for setting up the pipeline in concourse..

Since I dont have that many build,test,publish tasks I have a naming convention on them instead of loads of folders with 1-4 files in each.

The tree inside my pipeline folder in atom:

Note: the pipeline.yml file is still pretty long (~500 lines)

The Makefile, the ${} comes from the included contant-files:

#Setup Makefile constants for urls etc:
include ../Makefile.constants

#Setup Makefile constants for  repo:
include ./Makefile.constants


set-pipe:
    fly -t dev_refactor set-pipeline \
        --config pipeline.yml \
        --pipeline ${PIPELINE} \
        --var "client-repo=${CLIENT_REPO_URI}" \
        --var "client-branch=${CLIENT_BRANCH}" \
        --var "server-repo=${SERVER_REPO_URI}" \
        --var "server-branch=${SERVER_BRANCH}" \
        --var "private-key=$$(cat ~/.ssh/id_rsa_no_passphrase)" \
        --var "docker-registry=${DOCKER_REGISTRY}" \
        --var "docker-registry-cert=$$(cat ../keys/docker-registry/docker-registry.crt)" \
        --var "docker-registry-server-dist=${DOCKER_REGISTRY}/server" \
        --var "docker-registry-client-dist=${DOCKER_REGISTRY}/client" \
        --var "docker-registry-node=${DOCKER_REGISTRY}/node" \
        --var "docker-registry-maven=${DOCKER_REGISTRY}/maven" \
        --var "docker-registry-protractor=${DOCKER_REGISTRY}/protractor" \
        --var "docker-registry-npm-cache=${DOCKER_REGISTRY}/npm-cache" \
        --var "docker-registry-soap=${DOCKER_REGISTRY}/soap-ui" \
        --var "reports-server=${REPORTS_SERVER}" \
        --var "concourse-url=${CONCOURSE_URL}" \
        --var "nexus-url=${NEXUS_URL}"

.PHONY: set-pipe

destroy-pipe:
# Destroy the pipeline in concourse
    fly -t dev_refactor destroy-pipeline \
        --pipeline ${PIPELINE}
.PHONY: destroy-pipe