Trigger multiple builds parallel at one fire in Jenkins Pipeline

267 views Asked by At

Context

I have a few Jenkins pipelines which are based on the regex to select the matching jobs to run. The number of downstream jobs for each pipeline around 60.

So I have written down the declarative scripts to select these matching job then built it with Jenkins Build Plugin parallel. In general, it's something as below:

Finish deploy to A -->   Pipeline A (master node)
                            ->> 60 downstream jobs (slave nodes)

Finish deploy to B -->   Pipeline B (master node)
                            ->> 60 downstream jobs (slave nodes)

Finish deploy to C -->   Pipeline C (master node)
                            ->> 60 downstream jobs (slave nodes)

The master node is the server used to run Jenkins.

The slave nodes are the AWS EC2 instances used to run the task, my pool around 32 servers which can be used up to 180 tasks at once.

Expected

Let's say I have those pipeline triggered pipeline A, pipeline B, pipeline C in sequence then I expect the downstream jobs are triggered gonna be in the queue in sequence. It means first 60 jobs from A will be scheduled, built then so on pipeline B, pipeline C, given the slave executors are still free and available to use.

Observed

Even the AWS EC2 instances running no task (all available), the pipeline did not trigger all the jobs at one fire but part of its. It means a random number of jobs in pipeline A will be built first, then after a while, the rest will be built.

The pipeline script:

Stage('Integration Test Run') {
            steps {
                script {
                    matchingJobs = commonPipelineMethods.getTestJobs(venture_to_test, testAgainst)
                    parallel matchingJobs.collectEntries{downstreamJob-> [downstreamJob.name, commonPipelineMethods.buildSingleJob(this, downstreamJob)]}
                }
            }
        }
def buildSingleJob(steps, downstreamJob) {
    return {
        def result = steps.build job: downstreamJob.fullName, propagate: false
        steps.echo "${downstreamJob.fullName} finished: ${result.rawBuild.result}"
        }
    }

So I'm not sure if I need anything to config, setting up the pipeline script to get those downstream job running at one fire.

Could anybody please look into this and give me some suggestion? Thanks

0

There are 0 answers