I am working on a web backend / API provider that grabs realtime data from a 3rd party web API, puts it in a MySQL database and makes it available over an HTTP/JSON API.
I am providing the API with flask and working with the DB using SQLAlchemy Core.
For the realtime data grabbing part, I have functions that wrap the 3rd party API by sending a request, parsing the returned xml into a Python dict and returning it. We'll call these API wrappers.
I then call these functions within other methods which take the respective data, do any processing if needed (like time zone conversions etc.) and put it in the DB. We'll call these processors.
I've been reading about asynchronous I/O and eventlet specifically and I'm very impressed.
I'm going to incorporate it in my data grabbing code, but I have some questions first:
is it safe for me to monkey patch everything? considering I have flask, SQLAlchemy and a bunch of other libs, are there any downsides to monkey patching (assuming there is no late binding)?
What is the granularity I should divide my tasks to? I was thinking of creating a pool that periodically spawns processors. Then, once the processor reaches the part where it calls the API wrappers, the API wrappers will start a GreenPile for getting the actual HTTP data using eventlet.green.urllib2. Is this a good approach?
- Timeouts - I want to make sure no greenthreads ever hang. Is it a good approach to set the eventlet.Timeout to 10-15 seconds for every greenthread?
FYI, I have about 10 different sets of realtime data, and a processor is spawned every ~5-10 seconds.
Thanks!
It's safe to patch a module wrote by pure python and using standard lib.
Divide tasks by single fetch using green module as you can. Put the jobs into a queue, which also provided by eventlet, the every task worker fetch a job from the queue, then save the result into db after finish fetching, or send to a event.Event object to trigger the job which wait for the task finish.Or, both of the two processes.
UPDATED:
The eventlet official document strongly recommend use the patch at the fist line of the main module, and it's safe to call monkey_patch multiple times. Read more on page http://eventlet.net/doc/patching.html
There some green module can working with eventlet, all of them are in the eventlet.green. A list on bitbucket. Make sure use the green module in your code, or patch them before import 3th modules which use the standard libs.
But the monkey_patch only accept few module, it's necessary to import the green module manually.