I am currently trying to run Hadoop 2.6.0 on Amazon EC2 instances (multi-node cluster). I launch two Ubuntu 14.04 instances. One of them as master and the other as slave. The following is my configuration:
Master
-core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoopmaster:9000</value>
</property>
</configuration>
-hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/ubuntu/hadoop-2.6.0/hadoop_data/hdfs/namenode</value>
</property>
</configuration>
-yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoopmaster:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoopmaster:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoopmaster:8050</value>
</property>
-mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hadoopmaster:54311</value>
</property>
</configuration>
-masters
hadoopmaster
-slaves
hadoopslave1
Slave
-hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/ubuntu/hadoop-2.6.0/hadoop_data/hdfs/datanode</value>
</property>
</configuration>
Others are the same as master.
When I run hdfs namenode -format
it looks fine and exit with status 0. When I run start-all.sh
it gives the following output:
ubuntu@hadoopmaster:~/hadoop-2.6.0$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [hadoopmaster]
hadoopmaster: starting namenode, logging to /home/ubuntu/hadoop-2.6.0/logs/hadoop-ubuntu-namenode-hadoopmaster.out
hadoopslave1: starting datanode, logging to /home/ubuntu/hadoop-2.6.0/logs/hadoop-ubuntu-datanode-hadoopslave1.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/ubuntu/hadoop-2.6.0/logs/hadoop-ubuntu-secondarynamenode-hadoopmaster.out
starting yarn daemons
starting resourcemanager, logging to /home/ubuntu/hadoop-2.6.0/logs/yarn-ubuntu-resourcemanager-hadoopmaster.out
hadoopslave1: starting nodemanager, logging to /home/ubuntu/hadoop-2.6.0/logs/yarn-ubuntu-nodemanager-hadoopslave1.out
It sounds good and no error reported. However, when I run jps
on the master site, it gives the following output:
ubuntu@hadoopmaster:~/hadoop-2.6.0$ jps
3640 ResourceManager
3501 SecondaryNameNode
3701 Jps
The NameNode is missing! When I run jps
on slave site, I got the following
ubuntu@hadoopslave1:~/hadoop-2.6.0$ jps
1686 DataNode
1870 Jps
1817 NodeManager
Following is the log file for NameNode:
2015-06-11 04:16:18,987 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hadoopmaster/54.172.40.127
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.6.0
and the .out
file:
ulimit -a for user ubuntu
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 13357
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 13357
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I have repeated this several times and I got the same result. The NameNode is always missing. Could anyone give me some suggestion about this problem? Thank you so much!
Did you check the permissions for the folders under the path file:/home/ubuntu/hadoop-2.6.0/hadoop_data/hdfs/