I'm trying to distribute a function over an EMR cluster using Ray. The number of tasks(5,000) are much more than the number of available CPUs(512). My issue is that Ray is not distributing tasks to all the nodes fairly(PFA the Server Load Distribution from Ganglia). I do see from command line prints that its actually launching tasks in other nodes(also visible by the light green color in some nodes in the distribution below), but its way more focused on the master node(The red node in the distribution below). This is not very helpful as I'm getting better speeds using the multiprocessing library's parallelization in one large 96 core instance.
Any advice on what might be going wrong and how to make Ray fairly distribute tasks over all nodes and cores would be very helpful. Thanks!
Ray always try to schedule locally first in order to optimize the performance (and that’s why your master node has the highest load because your driver is running there normally). There are many ways to load balance workload across the cluster. For example, you can specify resources (like num_cpus, so that ray will schedule tasks on other nodes when there are not enough resources on the head node by other tasks) or use an API called placement group; https://docs.ray.io/en/master/placement-group.html
I also recommend you to post this on Github discussion so that you can get more techniques from other users :)! https://github.com/ray-project/ray/discussions