Google's RuntimeConfig API responds with 'Our systems have detected unusual traffic from your computer network'

Question

Since today (november 20 2018) we get error responses from Google's RuntimeConfig API: Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot... (check this link for complete HTML error)

We retrieve variables from Google's RuntimeConfig using the API in our code. We do quite a few request, but not more than before:

A developer starts his server locally, which retrieves all the needed variables (+- 30 everytime you start).

Requesting RuntimeConfig variables via GCloud results in the same HTML error: gcloud beta runtime-config configs variables get-value databaseHost --config-name database --project=your-test-environment

Other gcloud api requests work (projects describe, gsutil, etc).

How can I verify if I violated any terms? I can only find a usage limit in GCloud Console of 6000 calls per minute.

Answer 1

We had the exact same issue on November 20th when a large amount of our preemptibles were reallocated at the same time.

Our startup-scripts make use of the gcloud beta runtime-config...-commands and they all responded with 503. These commands responded correctly again after a few hours.

We have had a support-ticket with Google and there was a problem with their internal quota mechanisms at the time which since is fixed so the issue is resolved.

Answer 2

You can find the quotas for Runtime Configurator and how much of those you are using in the Cloud Console under IAM & Admin. In the Quotas section you can filter on Service = Cloud Runtime Configuration API and you should see all the quotas and how close to those you are for this API. There are 4 quotas that may affect you (docs here):

• 1200 Queries Per Minute (QPM) for delete, create, and update requests
• 600 QPM for watch requests
• 6000 QPM for get and list requests.
• 4MB of data per project, which consists of all data written to the Runtime Configurator service and accompanying metadata.