I just wrote an SSL supporting HTTP 1.1/1.0 proxy putting together some code for past two days. I used the HTTPWebResponse/HTTPWebRequest class to get the data from the Server. While relaying data from the server I send the headers first to the browser as soon as I have it and the then the response stream from the server. I noticed that when the response is Chunked if I just read using HTTPWebresponse.GetResponseStream() using a stream reader and forward it to the browser, the browser fails to load the page. After spending some time I realized that GetResponseStream() seems to be already de-chunked, so the browser fails to parse it (since the chunked response header is already sent to the browser which confused it). I made a workaround by removing the chunked header and then sending the responsestream together without chunking.
But I noticed that fiddlercore (royalty-free proxy library) somehow relays the chunked data without doing the workaround I did, it is written in .NET, so I suppose there should be a way to relay the chunks one by one.
My question is how to relay chucked response properly in a proxy when using streams? Also if my proxy is intended for the local machine does the performance would be poor if I sent the data together to the browser without chunking (while proxy uses chunking with the server and vice-versa when requested)?
I actually decided to chunk the data again each time I read in to the buffer from the responsestream (only if the response header is indicating a chunked response). I suppose the performance penalty in re-chunking would be negligible. If I just waited for all the chunks to be read as I mentioned in workaround, then there is a possibility that I would blow up the memory available to proxy.