Hadoop MapReduce (Yarn) using hosts with different power/specifications

Question

I currently have high power (cpu/ram) hosts in the cluster and we are considering to add some good storage but low power hosts.

My concern is that it will reduce the jobs performance. Map/Reducers from the new (less powerful) hosts will run slower and the more powerful ones will just have to wait for the result.

Is there a way to configure this in Yarn ? Maybe to set a priority for the hosts or to assign mapper/reducers according to the number of cores on each machines.

Thanks, Horatiu

Answer 1

In YARN you can provide a different hardware resources configuration to each of the cluster workers. The YARN will then decide how many tasks (containers) may be allocated at each worker to balance the workload.

In other words you can set different number of cores and memory in yarn-site.xml for each worker separately.

These two configuraton options are relevant for you:

• yarn.nodemanager.resource.cpu-vcores to set number of CPU cores per current worker node
• yarn.nodemanager.resource.memory-mb to set the amount of physical memory (in MB) that can be used at the current worker node