How to download via TIdHTTP parts from multiple servers?

Question

Currently I use TIdHTTP in Delphi 11 to automatically download updates of my software. My installer is relatively big (about 100 MB) and the download takes a couple of minutes or even more.

Supposing I have the same installation file on different servers, is there a way to use all these servers to improve the download speed, something like Torrent does?

Answer 1

Torrent works by downloading separate pieces of a file from multiple sources in parallel, and then putting the pieces together into the final file.

You can do that with TIdHTTP too, if all of the servers (or even just 1 server) support the HTTP Range request header. So, for example, you could download the file in 1KB chunks by downloading byte ranges 0-1023, 1024-2047, 2048-3071, and so on until the final chunk.

If your server(s) support that, then you can do the following:

1. Create a file on disk, and presize it to the total size of the final file (see How do you pre-allocate space for a file in C/C++ on Windows?).
2. For each piece of the file you want to download:
   • Create a TIdHTTP.
   • Create a TFileStream ¹ to the final file, requesting and sharing read/write access/rights, and then seek it to the desired start offset for the piece in the file.
   • Download the piece into the TFileStream, setting the TIdHTTP.Request.Range property to 'bytes=<start>-<end>', where start is the starting offset, and end is the ending offset, of the piece in the file.

¹ UPDATE: Oh wait, I forgot that TIdHTTP (more specifically, TIdIOHandler.ReadStream()) resizes the given TStream to the size of the data being downloaded, if that size is reported by the server (which it would be in this situation). You DON'T want that to happen when you have already presized the target file ahead of time, otherwise it will get truncated/corrupted if you download multiple TFileStreams into the same file. So, you can't use a standard TFileStream here. What you could do, though, is derive a new class from TFileStream and override its virtual SetSize() methods to do nothing. That should work, I think.

Alternatively:

1. For each piece you want to download:
   • Create a TIdHTTP.
   • Create a TFileStream (a standard TFileStream is fine here) to a separate temp file, requesting write-only access and sharing no rights, and do not seek it.
   • Download the piece to the TFileStream, setting TIdHTTP.Request.Range as described above.
2. When all pieces are downloaded, create a new file on disk and copy the bytes of each temp file, in order, into this final file (see How do I concatenate two files into one?), and delete the temp files.