How can I debug pauses in execution in the App Engine production environment?

Question

When looking at the Appstats graphs for my Google App Engine/Java application, I sometimes see gaps of seemly unexplainable time between RPCs. In the attached screenshot, there is a gap of almost 2 seconds between two calls to MemcacheService.get from a for-loop. This behavior does not appear when I run the code locally. How can I investigate the cause of anomalies such as this one, when my code is running in the App Engine cloud?

Apparently SO wont let me post images, so here's a link to the graph: http://i.imgur.com/bbCXr.jpg

Answer 1

Remember App Engine is shared hosting. While it doesn't seem to be anything near oversold shared VPS hosting you are still on a machine running other sites. If another sites code takes over some resources it is possible it will impact your applications performance. I am not fully aware of what resources App Engine is sensitive too, someone like @Nick-Johnson would be more able to tell you those. The way App Engine works it does prevent most of the common problems. When I look at that graph it just looks to me like another application had grabbed the machine for the gap.

Answer 2

While 2 seconds is an anomalously long time, unexplained gaps like this are usually down to one of two reasons: app code, or scheduling.

Since it sounds unlikely you're doing 2 seconds worth of processing work between those two calls, the most likely answer is scheduling: your app ran out its timeslice, and the OS scheduled other tasks to run for a while.