WebResponse contentLength property is -1?

Question

I have a windows application that gets URL and download .jpeg file. For some urls the ContentLength property is -1, therefore it throws exception.

Here is my code:

var url = new Uri(sUrlToReadFileFrom[i]);
_request = (HttpWebRequest)WebRequest.Create(url);
var response = (System.Net.WebResponse)_request.GetResponse();
_response = response;
_response.Close();

and Here is the url: http://photos.autonexus.com/imager/115-005-CA/P2GTPI4AB9/640/10132013133959/1FAHP0HA3AR373228_1.jpg and here is some information about my http request:

Headers = {Transfer-Encoding: chunked
Connection: keep-alive
Content-Disposition: inline; filename="phpThumb_generated_thumbnail.jpeg"
Content-Type: image/jpeg
Date: Wed, 12 Nov 2014 00:31:29 GMT
Server: nginx
X-Powered-By: PleskLin}

I think the chunked header cause the problem but I have been googling for 2 days and there is no good solution or I cant find a good one.

Here is the screen shot of the error:

As you see in line 130, because _response.ContentLength is already -1 therefore iSize will be -1 and it throws exception at line 149.

Answer 1

There is no requirement that a site provide a Content-Length header, and there's no guarantee that it will be correct. So you can't depend on it. If you try to use the ContentLength property value to allocate an array, or for any purpose than information, you're going to have trouble. It simply isn't reliable in the general case.

That's unfortunate, but you have to work around it. One solution is to create a MemoryStream. Then read blocks of data from the response stream and write them to the memory stream. Continue until the end of the response stream. Then get the `MemoryStream' buffer.

Kind of a pain, but it's the best you can do if the ContentLength isn't reliable.

For example: (Please note that I just tossed this off, so it might not be 100% working. But it should give you the idea.)

var response = (HttpWebResponse)request.GetResponse();
byte[] data; // will eventually hold the result
// create a MemoryStream to build the result
using (var mstrm = new MemoryStream())
{
    using (var s = response.GetResponseStream())
    {
        var tempBuffer = new byte[4096];
        int bytesRead;
        while ((bytesRead = s.Read(tempBuffer, 0, tempBuffer.Length)) != 0)
        {
            mstrm.Write(tempBuffer, 0, bytesRead);
        }
    }
    mstrm.Flush();
    data = mstrm.GetBuffer();
}
// at this point, the data[] array holds the data read from the stream.
// data.Length will tell you how large it is.

Answer 2

It's simple: don't use the content length header. It's perfectly normal if it is omitted. You should just read data as long as it is available. The server will keep sending it until it is done. You can use the Stream.CopyTo method, which will copy the response stream to a file or a MemoryStream or anything. That method also doesn't need to know the length, it just keeps reading until the end of the stream.

Update: code goes something like this

var resultStream = new MemoryStream()
using (var respStream = response.GetResponseStream())
    respStream.CopyTo(resultStream);
// now you can do anything with resultStream, like resultStream.ToArray()

Or you can write it to a file:

using (var fileStream = File.Create(@"d:\x\y.dat"))
using (var respStream = response.GetResponseStream())
    respStream.CopyTo(fileStream);

Answer 3

Just to contribute -- there is nothing wrong with the Content-Length header -- the server is setting it to -1 because it is returning your data 'Chunked' -- exactly as you surmised. Chunked data can be unlimited, and thus a Content-Length doesn't make sense. Technically the server still could return a length if it wanted to when it knows the aggregate size of the chunks it will send back, although I don't know if that is part of the spec.

The solutions above are fine for dealing with this -- but it has nothing to do with the Content-Length being unreliable -- it is as reliable as the service that provides it.