HttpListener : writing to outputstream slow depending on content?

Question

Removed the old question & rewriting completely because i've worked on this quite a bit to pinpoint the problem. My issue is that i'm writing a custom CMS with a custom server, with very very high speed/thoroughput as a goal, however i'm noticing that some data or data patterns cause major slowdowns (go from 0 to 55+ms response time). I really need someone better than me helping on this as i'm absolutely clueless about what is going on, i'm suspecting a bug in the .net Framework but i have no idea about where it could be, the little .net code browsing i did didn't suggest the output Stream does anything data-specific

Things that i've tested and am sure aren't the issue:

• Size of the content (larger content is faster)
• Type of the content (difference between the same content types)
• Most of the surrounding code (made a minimalist project to reproduce the bug, standing at around 15 lines, find the link at the bottom of the post, includes data to reproduce it, run it, test with 2 URL, see for yourself).
• Not an issue with webpages / cache etc, issue reproduced with a single image and CTRL+F5 in Firefox, removing the last few bytes of the image fixes it 100% of the time, adding them back causes the issue again
• Not an issue that exists outside of the outputstream (replacing it with a target memorystream doesn't show the issue)

How to reproduce the issue:

• Download & run the project
• Use your favorite browser and go to localhost:8080/magicnumber
• replace magicnumber in that url by what you want, you will receive the image back minus that amount of bytes

My result:

• Constant 50ms or so with that image
• Getting the magic number up to 1000 doesn't affect this at all
• a bit further (i think around 1080 ish?) it suddently drops to 0MS
• Not sure what is going on but it seems there are 2 requests per request at least when using CTRL+F5 in Firefox, in the correct case both are 0ms, in the error case the first remains 0ms but the other becomes 50ms, i'm assuming the first one is simply checking if the file cache is ok & i'm still answering but Firefox closes the connection or something?

Any help is much appreciated, placing all my rep on Bounty there as i really need to know if i go down this path / get more info to report this or if i go lower level and do my own http.sys interop (and, most of all, if the bug is only on the .net side or lower level & going lower level won't fix it!)

The sample file is a gziped array, my content is pre cached and pre compressed in db so this is representative of the data i need to send.

https://www.dropbox.com/s/ao63d7din939new/StackOverFlowSlowServerBug.zip

Edit : if i have fiddler open, the problematic test gets back to 0ms, i'm not sure what to make of it so far this means i'm getting a major slowdown, when sending some data, which isn't defined by the type of data but the actual data, and this doesn't happen if i have fiddler in between. I'm at loss!

Edit 2 : Tested with another browser just to be sure and, actually it's back to 0ms on IE so i'm assuming it may actually not be a HttpListener bug but instead a Firefox bug, i will edit my question & tags toward that if no one suggests otherwise. If this is the case, anyone know where i should be looking in FF's code to understand the issue? (it definately is an issue even if on their side since once again i'm comparing with 2 files, one larger than the other, same file format, the largest one always takes 0ms the smallest one always 55ms!)

Answer 1

Two requests problem:

Chrome:

• First request = favicon
• Second request = image

Firefox:

• First request = image for the tab
• Second request = image

More on this:

http://forums.mozillazine.org/viewtopic.php?t=341179

https://bugzilla.mozilla.org/show_bug.cgi?id=583351

IE:

• Only appears to make the one request

If you send the requests through fiddler you never get two coming through.

Performance problem:

Firstly there's a problem with the timer in your demo app. It's restarted everytime the async request handler fires, meaning that the timer started for request A will be restarted when request B is received, possibly before request A is finished, so you won't be getting the correct values. Create the stopwatch inside the ContinueWith callback instead.

Secondly I can't see anyway that "magicnumber" will really affect performance (unless it causes an exception to be thrown I guess). The only way I can cause performance to degrade is by issuing a lot of concurrent requests and causing the wait lock to be continually hit.

In summary: I don't think there's a problem with the HttpListener class