System.Net.HttpWebResponse.GetResponseStream() returns truncated body in WebException

Question

For some reason beyond my understanding requests made to a perticular website (https://learningnetwork.cisco.com/people/mrollins?view=profile) result in a reqsponse-object whose responsestream contain a trunkated version of the website.

The stream ends after 65536 bytes, which equals 2^16 bytes. I consider that a suspiciously round number. The requests complains of an internal server error, which I surpress, because I have already verified both that webbrowsers are able to make sense of this response, and that the full html is included in the response from the server. (using fiddler)

I've found the problem previously documented here, which is unsatisfactory for the simple reason that it ends on this note:

"I guess I'll have to hope the error doesn't exceed 65536 characters..."

Well it does.

Workarounds appreciated, or if anyone knows of an upcomming fix that is good too.

using System;
using System.IO;
using System.Net;
using System.Web.UI;

namespace Example
{
    public partial class _Default : Page
    {
        protected void Page_Load(object senderHidden, EventArgs eHidden)
        {
            //ServicePointManager.ServerCertificateValidationCallback += delegate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) { return true; };
            var cookieContainer = new CookieContainer();
            //Encoding enc = Encoding.UTF8;
            HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://learningnetwork.cisco.com/people/mrollins?view=profile");
            req.AllowAutoRedirect = false;

            req.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
            req.CookieContainer = cookieContainer;
            req.Accept = "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
            req.Method = "GET";
            req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12";
            req.KeepAlive = true;

            HttpWebResponse resp = null;
            try
            {
                resp = (HttpWebResponse)req.GetResponse();
            }
            catch (WebException e)
            {
                var r = e.Response as HttpWebResponse;
                var memstream = Read(r.GetResponseStream());
                var wrongLength = memstream.Length;
            }
        }

        public static MemoryStream Read(Stream stream)
        {
            MemoryStream memStream = new MemoryStream();

            byte[] readBuffer = new byte[4096];
            int bytesRead;
            while ((bytesRead = stream.Read(readBuffer, 0, readBuffer.Length)) > 0)
                memStream.Write(readBuffer, 0, bytesRead);
            return memStream;
        }
    }
}

Answer 1

HttpWebRequest has a static property that limits the length of web requests. this line of code, inserted before the request is made, solves the problem.

HttpWebRequest.DefaultMaximumErrorResponseLength = 1048576;