extra large file distribution

Question

I'm looking for ideas for infrastructure solution.

I have a file that is roughly between 400Mbs. I need it to be available to about 2500 computers. Those computer will download that file roughly on the same day over the course of 8 hours. Meaning they will not all access at the same second but rather over a span of 8 hours.

Would S3 be able to handle that load and if not, what would you suggeest a good configuration to distribute these files would be.

Bare in mind that most of the computers are in roughly the same geographical location so CDNing the content, while offshoring some of the load, will not solve the problem completely.

Thanks!

Answer 1

Looking for idea's? Well, here is 1: an important and often overlooked part of an infrastructure, are the in-between parts that connect the nodes said an experienced professional once to me.

He also said that high grade Cat-6 or Cat-7 UTP cables will in theory and in practice output more data at lower lags than Cat-5e cables. In other words: dont forget the hardware that is essentially carrying all the stuff. Ram, Raid, UTP, Switches, Routing etc. Once you have your Mb/s burst figured out, the rest should be worry free minutes awaiting the full transmissions completion.

PS. by "400Mbs" Do you mean 400 MB or 400 Mbits/s?

Answer 2

I would use the BitTorrent capabilities of S3 for this, see http://aws.amazon.com/s3/faqs/#What_is_the_BitTorrent_TM_protocol_and_how_do_I_use_it_with_Amazon_S3