Pipeline - Bitbucket -> Digital Ocean container registry -> deploy

Question

I need to make pipeline on Bitbucket repository which triggers on git push. This pipeline should create docker image push this image into the Digital Ocean container registry and then create Docker container for Kuberneties on the production server (Digital Ocean).

I have a pipeline but the step where pipeline push docker image to the DO container registry and pull it back is broken. There is a sleep(60) to wait while image will be registered and available. But it often pulls previouse old image. So I need to run pileline again and then it pull the right image from DO container registry. This is yaml file for Bitbucket pipeline

image: atlassian/default-image:3

options:
size: 1x

definitions:
services:
docker:
memory: 2048

pipelines:
#  custom:
#    dev-k8s:
#      - step:
#          name: Manual Build for Develop
#         services:
#           - docker
#          script
#            - docker build --build-arg VERSION=$VERSION_LABEL -t $IMAGE_TAG | tee -a logs.txt 2>&1
#          after-script:
#            - if [[ BITBUCKET_EXIT_CODE -eq 0 ]]; then exit 0; else echo "Step failed"; fi
#            - echo "Step failed"
#            - pipe: atlassian/slack-notify:2.1.0
#              variables:
#                WEBHOOK_URL: $SLACK_NOTIFICATION_URL_BACKEND
#                MESSAGE: '$(cat logs.txt)'
branches:
beta-k8s:
- step:
    name: Build for Beta & Push Image To Beta DigitalOcean
    services:
        - docker
    script:
        - set -o pipefail # important! makes sure the build fails when ANY of piped commands fail
        - export VERSION_LABEL=`[[ $BITBUCKET_TAG ]] && echo "$BITBUCKET_TAG-${BITBUCKET_COMMIT::7}" || echo "$BITBUCKET_REPO_FULL_NAME-${BITBUCKET_COMMIT::7}"`
        - export VERSION_REF=`[[ $BITBUCKET_TAG ]] && echo "$BITBUCKET_TAG" || echo "$BITBUCKET_BRANCH"`
        - export BUILD_NUMBER=`[[ $BITBUCKET_BUILD_NUMBER ]] && echo "$BITBUCKET_BUILD_NUMBER"`
        - cd deployment/kubernetes && docker build --build-arg VERSION=$VERSION_LABEL -t xxxxx-backend:beta-build-$BUILD_NUMBER -t xxxxx-backend:beta -f ./Dockerfile ../.. | tee -a logs.txt 2>&1
        - export REMOTE_IMAGE_TAG="registry.digitalocean.com/xxxxx/k8s-xxxxx-backend:beta" && echo REMOTE_IMAGE_TAG
        - export REMOTE_IMAGE_TAG_W_BUILD="registry.digitalocean.com/xxxxx/k8s-xxxxx-backend:beta-build-$BUILD_NUMBER" && echo REMOTE_IMAGE_TAG_W_BUILD
        - docker login -u $DIGITALOCEAN_CONTAINER_REGISTRY_TOKEN -p $DIGITALOCEAN_CONTAINER_REGISTRY_TOKEN registry.digitalocean.com | tee -a logs.txt 2>&1
        - docker tag xxxxx-backend:beta $REMOTE_IMAGE_TAG | tee -a logs.txt 2>&1
        - docker tag xxxxx-backend:beta $REMOTE_IMAGE_TAG_W_BUILD | tee -a logs.txt 2>&1
        - docker push $REMOTE_IMAGE_TAG | tee -a logs.txt 2>&1
        - docker push $REMOTE_IMAGE_TAG_W_BUILD | tee -a logs.txt 2>&1
        - echo "Waiting for a minute..."; sleep 60
- step:
    name: Refresh Beta Kubernetes
    trigger: automatic  # This is the problem. Should run on hook I think
    script:
        - pipe: atlassian/kubectl-run:3.6.0
          variables:
                KUBE_CONFIG: $KUBE_CONFIG
                KUBECTL_COMMAND: 'rollout restart deployment backend-beta backend-queue-default-beta backend-queue-notifications-beta'

Does anybody know how to fix this isue? I search for some hooks on DO container register but without success.

Answer 1

To fix the issue where the Kubernetes cluster is pulling an old image, you might want to use a more robust method than just a sleep(60). That method is not reliable as it assumes that the image will always be available after 60 seconds, which may not be the case.

+-----------------------------------------------+
|         Bitbucket Repository                  |
| +------------------+   +--------------------+ |
| | Trigger Pipeline |-->| Build Docker Image | |
| +------------------+   +--------------------+ |
|          |                                    |
|          v                                    |
| +------------------+   +--------------------+ |
| | Push to DO Reg.  |-->| Kubernetes Deploy  | |
| +------------------+   +--------------------+ |
+-----------------------------------------------+

Instead, you can use the image digest as a way to make sure Kubernetes pulls the exact image that was pushed.

After pushing the image to the Digital Ocean Container Registry, get the image digest from the push output.
Then, use the image digest to update the Kubernetes deployment, which will guarantee that the exact image is pulled.

The script section from your bitbucket-pipelines.yml file would be:

script:
    - set -o pipefail
    # (previous steps remain unchanged)
    - docker push $REMOTE_IMAGE_TAG | tee -a logs.txt 2>&1
    - export IMAGE_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' $REMOTE_IMAGE_TAG)
    - echo "Image digest is $IMAGE_DIGEST"
    # Use the image digest to update the Kubernetes deployment
    - pipe: atlassian/kubectl-run:3.6.0
      variables:
        KUBE_CONFIG: $KUBE_CONFIG
        KUBECTL_COMMAND: "set image deployment/backend-beta backend-beta=$IMAGE_DIGEST --record"

No need to wait after pushing the image, as you directly reference the image by its digest, which is immediately available after the push.
Do replace backend-beta in the kubectl set image command with the actual name of your deployment in Kubernetes.

Your updated pipeline process would be:

+-------------------------------------------------------------+
|                    Bitbucket Repository                     |
| +------------------+   +---------------------+   +--------+ |
| | Trigger Pipeline |-->| Build & Push Image  |-->| Deploy | |
| +------------------+   +---------------------+   +--------+ |
|          |                              |                   |
|          |                              |                   |
|          |                              v                   |
|          |            +---------------------------+         |
|          +----------->| Update K8s Deployment with|         |
|                       | Image Digest              |         |
|                       +---------------------------+         |
+-------------------------------------------------------------+