Application is taking data from mysql, processes it and puts back. It works on PHP & MySQL.
App runs on cron jobs, process_launcher cron job is scheduled to run every minute, and it starts 500 other php processes (threads) and those processes are actually choosing the data to process (1 piece of data = 1 row) and update it after it's done. processing the data takes 40-50 seconds per thread.
Now I'm moving this application on the cluster with 2 load balancers, 2 processing servers and 2 database servers (all the pairs are master-slave) and need some ideas on how load balancer should work.
I'm thinking to move the process_launcher on the load balancers (it'll be launched as cron job on both) and make it start the threads on randomly chosen processing server (maybe not random but on the one that has less load, but I'm not sure yet how to detect the server with least load quickly enough)
Any better ideas would be greatly appreciated.