Programmatically preventing exhaustion of allocated Google App Engine Quotas

Question

Am working on a gae app using python. The app involves some crowd-sourced data collection system and data used in the app is submitted by users all-over the country. Now, am using the default quotas (Free) but am faced with a problem of ensuring at least 99% up-time for my app.

The challenge is that Google blocks any further requests being routed to your app once you exhaust your allocated quotas, and during a recent testing spree, one person was able to build an automated posting script that quickly exhausted the CPU quota - after that, the app would only serve HTTP 403 Forbidden status code for the request instead of calling a request handler. Now, I have patched the system not to allow automated postings, but how can I guarantee that human users don't cause a similar "blackout" at production time?

I know of the Quota API, but am thinking that can only give me profiling info for my app, I want a way of slowing down the rate of requests (e.g per minute for the per minute quotas) without serving error pages or blackouts.

Any suggestions?

Answer 1

One common solution of this problem is to delegate the tasks to a rate limited taskqueue.

For example:

queue:
- name: mail-throttle
  rate: 2000/d
  bucket_size: 10
- name: background-processing-throttle
  rate: 5/s

In this way you can control the usage of all the parts of your application forcing them to stay in the range of the available quotas.

A couple of caveats:
1. Queues deliver a best effort FIFO order
2. Enqueuing/Execution of a task counts toward several quotas