Why is database connection in Google Cloud Build slow?

Question

I'm running a YII project that uses Codeception for unit and integration testing. In my Dockerfile, the last step of my build is to run the integration test with RUN ./vendor/bin/codecept run

I have already set up CloudSQL and whitelisted the IP address so that CloudBuild can establish a connection to the DB.

In the cloudbuild.yaml, I have the first step to build the docker image:

  - name: gcr.io/cloud-builders/docker
    args:
      - build
      - '--no-cache'
      - '-t'
      - '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
      - .
      - '-f'
      - Dockerfile
      - '--build-arg'
      - environmentId=$_GCPENVIRONMENTID
    id: Build

When the docker runs the integration test RUN ./vendor/bin/codecept run on CloudBuild, it is incredibly slow when it involves database connection. When I benchmark this against my local machine, while connecting to the same CloudSQL DB, it runs at least 10 times faster.

I've done everything I can to isolate the issue. CloudSQL runs fast. The Docker build is fast. But for some reason, Database connection from CloudBuild to CloudSQL is really slow. Why is that?

CloudSQL configuration

• Database: MySQL 8.0.18
• Cores: 1vCPU
• RAM: 614.4 MB
• Storage: SSD

Test Results

I ran 2 test run with 2 different database. In each test run, I run with Cloud Build and my own localhost. As control, I have test script that includes testing without database interaction.

The results are near identical, as expected from network connection. Both database are hosting in Singapore region.

Test 1 - Google Cloud SQL

• Cloud Build - 10 seconds to 1.5 minutes
• Localhost - 1 second to 4 seconds
• Cloud Build Controls - < 10ms
• Localhost Controls - < 50ms

Test 2 - Digital Ocean database

• Cloud Build - 10 seconds to 1.5 minutes
• Localhost - 1 second to 4 seconds
• Cloud Build Controls - < 10ms
• Localhost Controls - < 50ms

Answer 1

I had word with Google Support. It seems that all Cloud Build triggers is running on us-central1. Connection to regions outside of us-central1 has a network performance penalty. As for my case, since my database is located in Singapore region, I experience severe performance penalty.

There is currently a way to port all Cloud Build to other selected regions. Unfortunately, this feature is still in beta and Singapore is not among the selected regions that can run Cloud Build.

If you're lucky enough to be in a region that support Cloud Build in beta, you may sign up to be in the wait-list at https://docs.google.com/forms/d/e/1FAIpQLSdAkLtvPUQFCgAUcHUi1VDHn_GaQz_sDOvFxpg3JXIJdIZY6g/viewform

This answer isn't exactly a solution, but I hope it can provide some insights to the cause of the problem.

Answer 2

I think Cloud Build is network-limited (at least to Cloud SQL) to ~16kbps ingress and ~2kbps egress. I get these numbers by noting that no matter what instance class I use for my Cloud SQL MySQL instances, when I run unit testing, I see different CPU and Memory usage, yet same network traffic amount. I've also tried this using different Cloud Build machine-type values and even large-CPU/large-Memory types result in the same test-time (slow).