How to create and modify Kubernetes resources across multiple clusters as part of a Jenkins build process?

Question

Specifically, there are several resources (currently defined with YAML) in existing Kubernetes clusters that I would like to modify as part of the build process, and some others that I would like to create from scratch. In each case, I want to do so in multiple regions to keep all regions in sync.

The resources in question are Agones fleets, which look like this (actual values removed, but representative):

apiVersion: agones.dev/v1
kind: Fleet
metadata:
  annotations:
    agones.dev/sdk-version: 1.11.0
  name: test
  namespace: game-servers
  resourceVersion: "12324578"
  selfLink: /apis/agones.dev/v1/namespaces/game-servers/fleets/test
spec:
  replicas: 1
  scheduling: Packed
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        branch: test
        git_commit: 1b12371827fdea31231230901876ffe
    spec:
      health:
        disabled: false
        failureThreshold: 5
        initialDelaySeconds: 10
        periodSeconds: 5
      ports:
      - containerPort: 1234
        name: default
        portPolicy: Dynamic
        protocol: UDP
      sdkServer:
        logLevel: Info
      template:
        metadata:
          creationTimestamp: null
          labels:
            role: game-servers
        spec:
          containers:
          - image: registry.example.com/gameserver:1b12371827fdea31231230901876ffe
            name: agones
            resources:
              limits:
                cpu: 500m
                memory: 512m
              requests:
                cpu: 100m
                memory: 256Mi
          nodeSelector:
            role: game-servers

Where there is an existing fleet, I would like to inject the latest git commit into the labels, as well as the image to use (assume it will be tagged appropriately on the registry).

If there is no existing fleet, for certain values I would like to to loop through and create new fleets from scratch, with similar characteristics as above. I have tried a couple of different ways and failed - from issues with permissions on clusters to odd errors when attempting to use pretty straight forward for loops in Jenkins/Groovy.

Answer 1

As with many things Jenkins related, to make this work a couple of plugins will be needed, specifically:

• Kubernetes CLI Plugin
• snakeyaml Plugin (this is per the readYaml Jenkins docs)

Assumptions:

• These actions will be run on Linux based containers configured correctly for executing Jenkins builds
• Several required "variables" will be read from the local file system - these may be build artifacts from earlier in a pipeline, or ENV variables, depending on the setup
• Kubernetes service accounts already exist on the clusters with sufficient permissions to perform the tasks
• Credentials for those accounts have been configured in Jenkins
• fleetconfig.yaml is available on the local filesystem and is a complete fleet config similar to the one posed in the question

In order to treat the various fleets differently, some selection criteria will need to be applied based on the name of the fleet itself, then a loop is needed to go through each region etc.

To keep this simple, there will be a basic if statement to select between types (this could easily be expanded), and then a 2 element loop to go loop through more than one region (again easily expandable to more).

This is written as a complete Stage in Jenkins terms, but obviously is not completely executable on its own. It is as close as it can be to an existing, tested, working configuration that has been run multiple times daily for quite some time.

Although this sticks to the example resources, there is no reason it could not be used to modify, create other resources in Kubernetes generally.

stage('DeployFleets') {
  agent {
    node {
      label 'k8s-node-linux'
    }
  }
  steps {
    script {
      // assume we can read the Fleet name in from a file
      FLEET_NAME = readFile("/path/to/FLEET_NAME")
      // let's assume that test is one of the fleets that is being modified, not created, deal with that first
      container('jenkins-worker'){
        if (FLEET_NAME != 'test') {
          script {
            // again, assume we can read the commit from a file
            def GIT_COMMIT_TAG = readFile("/path/to/GIT_COMMIT")
            //  create a map of 2 fictional regions with an account to use and an cluster adddress for that region
            def DEPLOY_REGIONS = [
              "us-east-1": ["jenkins_service_acct_use1", 'https://useast1-cluster.example.com'],
              "us-east-2": ["jenkins_service_acct_use2", 'https://useast2-cluster.example.com'],
            ]
            // this each construction is needed in order to get around https://issues.jenkins-ci.org/browse/JENKINS-49732 which prevents using a for(element in DEPLOY_REGIONS)
            DEPLOY_REGIONS.each { element ->
              withKubeCredentials([[credentialsId: element.value[0], serverUrl: element.value[1]]]) {
                sh """
                kubectl patch fleet ${FLEET_NAME} -n game-servers --type=json -p='[{"op": "replace", "path": "/spec/template/spec/template/spec/containers/0/image", "value":"registry.example.com/gameserver/${FLEET_NAME}:${GIT_COMMIT_TAG}"}]'
                kubectl patch fleet ${FLEET_NAME} -n game-servers --type=json -p='[{"op": "replace", "path": "/spec/template/metadata/labels/git_commit", "value":"${GIT_COMMIT_TAG}"}]'
                """
              }
            }
          }
        } else {
          // rather than patching here, create a fleet from scratch using a source YAML file as a template
          script {
            def GIT_COMMIT_TAG = readFile("/path/to/GIT_COMMIT")
            def NUM_REPLICAS = 1
            def DEPLOY_REGIONS = [
              "us-east-1": ["jenkins_service_acct_use1", 'https://useast1-cluster.example.com'],
              "us-east-2": ["jenkins_service_acct_use2", 'https://useast2-cluster.example.com'],
            ]
            // see note above about each construct
            DEPLOY_REGIONS.each { element ->
              // assume template available on file system
              def FLEET_CONFIG = readYaml file: "/path/to/fleetconfig.yaml"
              FLEET_CONFIG.metadata.name = env.SOME_SANE_NAME
              FLEET_CONFIG.spec.template.metadata.labels.git_commit = GIT_COMMIT_TAG
              FLEET_CONFIG.spec.replicas = NUM_REPLICAS
              FLEET_CONFIG.spec.template.spec.template.spec.containers[0].image = "registry.example.com/gameserver/${FLEET_NAME}:${GIT_COMMIT_TAG}"
              writeYaml file: "${env.SOME_SANE_NAME}_fleet.yaml", data: FLEET_CONFIG, overwrite: true
              withKubeCredentials([[credentialsId: element.value[0], serverUrl: element.value[1]]]) {
                sh """
                kubectl -n game-servers apply -f "${env.SOME_SANE_NAME}_fleet.yaml"
                """
              }
            }
          }
        }
      }
    }
  }
}

This is not particularly difficult given the YAML manipulation utilities given to us in Jenkins via the plugin but finding an approach that works from end to end can be a challenge. The use of the patch command in kubectl makes the patching less "native" to Jenkins, but the convenience is more than worth it (an alternative would be using the REST API instead for example). The foreach structure looks odd, but is needed to avoid a long-standing bug in Jenkins.