Maven lifecycle within jenkins pipeline - how to best separate responsibilities?

Question

When working with jenkins 2 (declarative) pipelines and maven I always have a problem with how to organize things within the pipeline to make it resusable and flexible.

On the one side I would like to seperate the pipepline into logical stages like:

pipeline
 {
  stages
   {
    stage('Clean') {}
    stage('Build') {}
    stage('Test') {}
    stage('Sanity check') {}
    stage('Documentation') {}
    stage('Deploy - Test') {}
    stage('Selenium tests') {}
    stage('Deploy - Production') {}
    stage('Deliver') {}
   }
 }

On the other hand I have maven which runs with

mvn clean deploy site

Simply I could split up maven to

mvn clean
mvn deploy
mvn site

But the 'deploy' includes all lifecycle phases from

• validate
• compile
• test
• package
• verify
• install
• deploy

So I saw a lot of pipline examples which do things like

sh 'mvn clean compile'

and

sh 'mvn test'

which results in repeating the validate and compile step a second time and waste "time/resources" in this way. This could be resolved with doing a

sh 'mvn surefire:test'

instead of running the whole lifecycle again.

So my question is - which is the best way to get a good balance between the jenkins pipline stages and the maven lifecycle? For me I see two ways:

1. Split up the maven lifecycles to as much pipeline stages as possible - which will result in better jenkins user feedback (see which stage fails etc.)
2. Let maven do everything and use the jenkins pipeline only to work with the results of maven (i.e. analyzing unit test results etc.)

Or did I missunderstand something in the CI/CD practice?

Answer 1

Two month later I think I have a well balanced Jenkins pipeline script that is not complete, but works stable on windows and linux. It avoids pitfalls of other examples I have seen.

Jenkinsfile

pipeline
 {
  agent any

  tools
   {
    maven 'Maven3'
    jdk 'JDK8'
   }

  options
   {
    buildDiscarder(logRotator(numToKeepStr: '4'))
    skipStagesAfterUnstable()
    disableConcurrentBuilds()
   }


  triggers
   {
    // MINUTE HOUR DOM MONTH DOW
    pollSCM('H 6-18/4 * * 1-5')
   }


  stages
   {
    stage('Clean')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode clean'
           }
          else
           {
            bat 'mvn --batch-mode clean'
           }
         }
       }
     }

    stage('Build')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode compile'
           }
          else
           {
            bat 'mvn --batch-mode compile'
           }
         }
       }
     }

    stage('UnitTests')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode resources:testResources compiler:testCompile surefire:test'
           }
          else
           {
            bat 'mvn --batch-mode resources:testResources compiler:testCompile surefire:test'
           }
         }
       }
      post
       {
        always
         {
          junit testResults: 'target/surefire-reports/*.xml'
         }
       }
     }

    stage('Sanity check')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode checkstyle:checkstyle pmd:pmd pmd:cpd com.github.spotbugs:spotbugs-maven-plugin:spotbugs'
           }
          else
           {
            bat 'mvn --batch-mode checkstyle:checkstyle pmd:pmd pmd:cpd com.github.spotbugs:spotbugs-maven-plugin:spotbugs'
           }
         }
       }
     }

    stage('Packaging')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode jar:jar'
           }
          else
           {
            bat 'mvn --batch-mode jar:jar'
           }
         }
       }
     }

    stage('install local')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode jar:jar source:jar install:install'
           }
          else
           {
            bat 'mvn --batch-mode jar:jar source:jar install:install' // maven-jar-plugin falseCreation default is false, so no doubled jar construction here, but required for maven-install-plugin internal data
           }
         }
       }
     }

    stage('Documentation')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode site'
           }
          else
           {
            bat 'mvn --batch-mode site'
           }
         }
       }
      post
       {
        always
         {
          publishHTML(target: [reportName: 'Site', reportDir: 'target/site', reportFiles: 'index.html', keepAll: false])
         }
       }
     }

    stage('Deploy test')
     {
      steps
       {      
        script
         {
          if (isUnix()) 
           {
            // todo
           }
          else
           {
            bat returnStatus: true, script: 'sc stop Tomcat8'
            sleep(time:30, unit:"SECONDS")
            bat returnStatus: true, script: 'C:\\scripts\\clean.bat'
            bat returnStatus: true, script: 'robocopy "target" "C:\\Program Files\\Apache Software Foundation\\Tomcat 9.0\\webapps" Test.war'
            bat 'sc start Tomcat8'
            sleep(time:30, unit:"SECONDS")
           }
         }
       }
     }

    stage('Integration tests')
     {
      steps
       {
        script
         {
          if (isUnix()) 
           {
            sh 'mvn --batch-mode failsafe:integration-test failsafe:verify'
           }
          else
           {
            bat 'mvn --batch-mode failsafe:integration-test failsafe:verify'
           }
         }
       }
     }

   }

 }

Hopefully this is interesting for other developers outside there.

I will update this here when I significantly improve it over time.

For those who also wish to see a maven pom along with a Jenkinsfile please have a look at my small example project at github: TemplateEngine

Answer 2

I think there is no right answer, but the following example worked for us.

stage('Build and Unit Test') {
    mvn clean deploy -> with unit tests, without integration tests, deploy local

    deploy local:
    You can define in a maven profile the distributionManagement like:
    <distributionManagement>
        <repository>
            <id>localFile</id>
            <url>file:target/repository/</url>
        </repository>
        <snapshotRepository>
            <id>localFile</id>
            <url>file:target/repository/</url>
        </snapshotRepository>
    </distributionManagement>
}   

stage('Pre Integration Tests') {
    The binaries are now in target/repository.
    From there you can use the binaries as you like.
    Copy them to a server, deploy them on an application server, etc.
}

stage('Integration Tests') {
    maven failsafe:integration-test failsafe:verify
    Already all tests are compiled, just execute them and verify the result.
}

stage('Deploy to Binary Repository (Nexus, Artifactory, etc)') {
    Now if everything is ok, finally upload the Binaries.
    For that we use wagon-maven-plugin
    So from target/repository the files are uploaded to the Binary Repository.
}

So to wrap this up:

• Fail fast. If a unit test has errors -> fail the build.
• Only build once. Use the same binaries for test, deployment/integration test, upload to repository, etc.
• With that the stages are logical units, which gives you enough feedback where to look for errors.