Downloading data with broken header

Question

I just got the following problem:

I need to download data which is behind a login page. When I however make my get request, the server is providing bad data - the content is there, but no content length in header set, its an empty field. I looked it up with Fiddler, and its the same when I try to download the file with browser, but browser completes the download ok, while C# drops with exception when getting the response object from my request.

The header looks like this :

HTTP/1.1 200 OK
Date: Sat, 06 Dec 2014 11:55:06 GMT
Server: Apache
Content-Disposition: attachment; filename=;
Pragma: no-cache
Expires: 0
Content-Length: 
X-Powered-By: PleskLin
Connection: close
Content-Type: application/octet-stream

 Hersteller;"Hersteller Art-Nr";"Lieferant Art-Nr";Ma�stab;Bezeichnung;EAN;"EK (netto)";UVP;USt;Verkaufseinheit;Hinweis;"Letzte Pro...

My code looks like this

    public string ReadPage(string path, string method = "GET"){
        var result = "";
        var request = (HttpWebRequest)WebRequest.Create(path);
        request.Method = method;            
        request.Host = "somehost.de";
        request.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.NoCacheNoStore);
        request.Referer = @"http://somehost.de/login.php?redir=list.php%3Ftype%3Dmm";
        request.AllowAutoRedirect = true;
        request.Headers.Add("Cookie", LoginCookie);
        try
        {
        var response = request.GetResponse();           
            using (StreamReader sr = new StreamReader(response.GetResponseStream()))
            {
                result = sr.ReadToEnd();
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
            //throw;
        }
        return result;            
    }

The exception appears in the var response = request.GetResponse(); line. Any idea how to fix this problem? I just want it to carry on and let me read out the data.

Forgot the exception - its a WebException with a message The server committed a protocol violation. Section=ResponseHeader Detail='Content-Length' header value is invalid

Answer 1

I'm really loving the answer of Matthew: .Net HttpWebRequest.GetResponse() raises exception when http status code 400 (bad request) is returned

litte bit changed

public static class WebRequestExtensions
{
    public static WebResponse GetWEResponse(this WebRequest request)
    {
        if (request == null)
        {
            throw new ArgumentNullException("request");
        }

        try
        {
            return request.GetResponse();
        }
        catch (WebException e)
        {
            return e.Response;
        }
    }
}

You can use this now by using request.GetWEResponse

public string ReadPage(string path, string method = "GET") {
    var result = "";
    var request = (HttpWebRequest)WebRequest.Create(path);
    request.Method = method;            
    request.Host = "somehost.de";
    request.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.NoCacheNoStore);
    request.Referer = @"http://somehost.de/login.php?redir=list.php%3Ftype%3Dmm";
    request.AllowAutoRedirect = true;
    request.Headers.Add("Cookie", LoginCookie);
    try
    {
        var response = request.GetWEResponse();           
        using (StreamReader sr = new StreamReader(response.GetResponseStream()))
        {
            result = sr.ReadToEnd();
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
        //throw;
    }
    return result;            
}

or better

public string ReadPage(string path, string method = "GET")
{
    var request = (HttpWebRequest)WebRequest.Create(path);

    request.Method = method;            
    request.Host = "somehost.de";
    request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
    request.Referer = @"http://somehost.de/login.php?redir=list.php%3Ftype%3Dmm";
    request.AllowAutoRedirect = true;
    request.Headers.Add("Cookie", LoginCookie);

    using (var response = (HttpWebResponse)request.GetWEResponse())
    using (var reader = new StreamReader(response.GetResponseStream()))
    {
        return reader.ReadToEnd();
    }
}