Getting HTTP range right, so that Down Them All works

1.3k views Asked by At

I built web app which is supposed to serve Recorded TV as a download (For the folks yelling out that that is illegal, I can tell them: in Germany it isn't, if you do it right). You can have a look at the project at http://niklasfi.de:8080.

Now about my problem: Download + Resume works with plain firefox. But i am experiencing problems with some download managers. So I am wondering why. I think this has to do with me getting the HTTP wrong or something.

When a client wants to download something, it is first given a Ticket (In our case this is 45729410) and redirected to the actual download link.

GET /createTicketFor/Matrix_Revolutions_11.02.13_22-30_pro7_155_TVOON_DE.mpg.avi.otrkey HTTP/1.1
Host: niklasfi.de:8080
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en;q=0.7,en-gb;q=0.5,en-au;q=0.3,en-us;q=0.2
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://niklasfi.de:8080/

HTTP/1.1 302 Moved Temporarily
location: /download/45729410
Connection: keep-alive
Transfer-Encoding: chunked

Afterwards firefox sends a get request to the download link (this is done while you are provieded with the download options (open with, download, downThaOneClick)

GET /download/45729410 HTTP/1.1
Host: niklasfi.de:8080
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en;q=0.7,en-gb;q=0.5,en-au;q=0.3,en-us;q=0.2
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://niklasfi.de:8080/

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=Matrix_Revolutions_11.02.13_22-30_pro7_155_TVOON_DE.mpg.avi.otrkey
Content-Length: 1428013574
Connection: keep-alive

I chose the latter, so Down them All starts 4 downloads:

=== DTA.1 ===
GET /download/45729410 HTTP/1.1
Host: niklasfi.de:8080
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, application/metalink4+xml;q=0.9,application/metalink+xml;q=0.8
Accept-Language: de-de,de;q=0.8,en;q=0.7,en-gb;q=0.5,en-au;q=0.3,en-us;q=0.2
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close
Referer: http://niklasfi.de:8080/
Want-Digest: SHA512;q=0.9, SHA256;q=0.7, SHA, SHA1;q=0.5, MD5;q=0.3
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=Matrix_Revolutions_11.02.13_22-30_pro7_155_TVOON_DE.mpg.avi.otrkey
Content-Length: 1428013574
Connection: close

=== DTA.2 ===
GET /download/45729410 HTTP/1.1
Host: niklasfi.de:8080
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en;q=0.7,en-gb;q=0.5,en-au;q=0.3,en-us;q=0.2
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close
Range: bytes=714610068-
Referer: http://niklasfi.de:8080/
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=Matrix_Revolutions_11.02.13_22-30_pro7_155_TVOON_DE.mpg.avi.otrkey
Content-Length: 1428013574
Content-Range: bytes 714610068-1428013573/1428013574
Connection: close

=== DTA.3 ===
GET /download/45729410 HTTP/1.1
Host: niklasfi.de:8080
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en;q=0.7,en-gb;q=0.5,en-au;q=0.3,en-us;q=0.2
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close
Range: bytes=357908315-
Referer: http://niklasfi.de:8080/
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=Matrix_Revolutions_11.02.13_22-30_pro7_155_TVOON_DE.mpg.avi.otrkey
Content-Length: 1428013574
Content-Range: bytes 357908315-1428013573/1428013574
Connection: close

=== DTA.4 ===
GET /download/45729410 HTTP/1.1
Host: niklasfi.de:8080
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en;q=0.7,en-gb;q=0.5,en-au;q=0.3,en-us;q=0.2
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close
Range: bytes=1071311822-
Referer: http://niklasfi.de:8080/
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=Matrix_Revolutions_11.02.13_22-30_pro7_155_TVOON_DE.mpg.avi.otrkey
Content-Length: 1428013574
Content-Range: bytes 1071311822-1428013573/1428013574
Connection: close

What makes me wonder is that the Range headers of all requests do not give an end-point. DTA shoule have the inforation needed since it "knows" where it has to start the other three requests.

So far so good. Everything worked. But no! DTA cancels the last 3 requests. Why is that?

This is not too bad since one can still pause the Download which is what I want. In fact lets do that now. I click the "Pause" button. On resume the following two requests are sent:

=== RESTART ===
GET /download/45729410 HTTP/1.1
Host: niklasfi.de:8080
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en;q=0.7,en-gb;q=0.5,en-au;q=0.3,en-us;q=0.2
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close
Range: bytes=10909760-
Referer: http://niklasfi.de:8080/
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=Matrix_Revolutions_11.02.13_22-30_pro7_155_TVOON_DE.mpg.avi.otrkey
Content-Length: 1428013574
Content-Range: bytes 10909760-1428013573/1428013574
Connection: close

=== RESTART +1 ===
GET /download/45729410 HTTP/1.1
Host: niklasfi.de:8080
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110216 Firefox/4.0b12pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, application/metalink4+xml;q=0.9,application/metalink+xml;q=0.8
Accept-Language: de-de,de;q=0.8,en;q=0.7,en-gb;q=0.5,en-au;q=0.3,en-us;q=0.2
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close
Referer: http://niklasfi.de:8080/
Want-Digest: SHA512;q=0.9, SHA256;q=0.7, SHA, SHA1;q=0.5, MD5;q=0.3
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=Matrix_Revolutions_11.02.13_22-30_pro7_155_TVOON_DE.mpg.avi.otrkey
Content-Length: 1428013574
Connection: close

But no! Suddenly the whole download up until here is truncated and I can not even pause my download any more. The only option I have is to cancel it. And I still only uses one download "thread".

I would be glad if you posted any Ideas explaining this behaviour. (Thanks for reading this far anyway)

1

There are 1 answers

0
niklasfi On BEST ANSWER

instead of 200, 206 has to be sent as a response if 'Partial Content' is provided