Bit of an odd problem here. We're migrating a website to a new software platform. As part of this migration, we must copy files from one Amazon S3 bucket to another. There are hundreds of thousands of files.
We also must use the software we have (phpFox) to do this. Basically a php framework.
The job is broken in to segments that we call using an offset in the URL.
Basically: Copy 10 files and update the database as necessary Increase offset by 10 Rinse, repeat.
The API traffic is light, the load on the server is sub 1%, however, if we open more than two tabs on any one machine to the server, the script begins slowing down proportionally, as if the web server (Apache) is queuing the commands instead of running them in parallel.
We've found that if we open two tabs on many machines, it scales up as expected. In order to either saturate our uplink or put any noticeable load on the server, we need to fill the room with laptops. While comical, this is also highly impractical and generally a pain in the ass. There has gotta be a better way here.
I've tried increasing the max spare processes, and requests per child, etc etc 10 fold and there was no noticeable increase in speed. What are we missing? How do I nicely tell Apache to temporarily let anyone connect as many times as they like and go nuts?
PHP 5.3.8 PHP Sapi: apache2handler PHP loaded extensions: Core date ereg libxml openssl pcre sqlite3 zlib bcmath calendar ctype curl dom fileinfo filter ftp gd hash iconv SPL json mbstring mysql session standard posix Reflection Phar SimpleXML sockets SQLite imap tokenizer xml xmlreader xmlwriter zip apache2handler
MYSQL: MySQL 5.0.92-community
Thanks for reading!
Most web browsers have a limit on the number of concurrent connections they'll make to a given web server. I'm not sure about all browsers, but one browser I know that it can be configured for is Mozilla Firefox, using the Fasterfox extension:
https://addons.mozilla.org/en-US/firefox/addon/fasterfox-9148/
That being said -- is there really no way you can write a script to do the migration on the server, rather than running it through a web interface??