I'm running MR java program with Yarn , even though Number of mappers are 24 , but actual running mappers are 10 , the remaining 14 maps are in pending state. how to make them also to running mode . we are running MAPR 6 node cluster . changed the below properties in mapred-site.xml and Yarn-site.xml . these values are overridden from the default values , still I'm not seeing any perfomance Bench mark .
Note . I'm running same Program with MRv1 performance wise it's better some how . So please suggest me how to utilize the cluster utilization resources well .
command Used : yarn jar /opt/cluster/bin/logmessage-1.0-SNAPSHOT.jar com.message.WordPreprocessDriver -Dmapreduce.input.fileinputformat.split.maxsize=33554432 /data/123.txt
In yarn-site
<name>yarn.nodemanager.resource.memory-mb</name>
<value>20960</value>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>
In mapred-site
<name>mapreduce.map.memory.mb</name>
<value>4096</value>
<name>mapreduce.reduce.memory.mb</name>
<value>8192</value>
<name>mapreduce.map.java.opts</name>
<value>-Xmx3072m</value>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx6144m</value>
mapreduce.job.maps
<name> mapreduce.job.maps</name>
<value>4</value>
You see Actually the total containers were running as 11 . there was 40 vcores were available, among them 11 vcores were used while running MR program . can you please share what properties needs to change for that .?
Thank you , Madhu