TechQA.

my datanode is not starting in hadoop 2.7.3 multi nodes

1.2k views Asked by At

my datanode is not starting in hadoop 2.7.3 multi nodes ( 1master, 2 slaves)

Here are my configuration files :

core-site.xml ( in master and slaves)

<configuration>
  <property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://Hadoop:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>

mapred-site.xml (in master and slaves)

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>Hadoop:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

hdfs-site.xml (in master)

<configuration>
<property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/var/lib/hadoop/hdfs/namenode</value>
  </property>
</configuration>

hdfs-site.xml (in slaves)

<configuration>
<property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/var/lib/hadoop/hdfs/datanode</value>
  </property>
</configuration>

yarn-site.xml ( in master and slaves)

<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>Hadoop:8025</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>Hadoop:8035</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>Hadoop:8050</value>
</property>

jps in master node :

13856 SecondaryNameNode
14083 Jps
13620 NameNode
14010 ResourceManager

jps in slaves

6162 Jps
6044 NodeManager

log file in slave 1

 root@ubuntu:/usr/local/lib/hadoop-2.7.3/logs# gedit hadoop-root-datanode-ubuntu.log
2016-12-24 05:28:42,854 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = ubuntu/127.0.1.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.7.3
STARTUP_MSG:   classpath = /usr/local/lib/hadoop-2.7.3/etc/hadoop:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/asm-3.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/xz-1.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/activation-1.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/junit-4.11.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/hadoop-nfs-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/usr/local/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z
STARTUP_MSG:   java = 1.8.0_111
************************************************************/
2016-12-24 05:28:42,881 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2016-12-24 05:28:44,573 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2016-12-24 05:28:44,737 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2016-12-24 05:28:44,737 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2016-12-24 05:28:44,743 INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576
2016-12-24 05:28:44,745 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is ubuntu
2016-12-24 05:28:44,761 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode with maxLockedMemory = 0
2016-12-24 05:28:44,826 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:50010
2016-12-24 05:28:44,828 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
2016-12-24 05:28:44,828 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 5
2016-12-24 05:28:45,010 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-12-24 05:28:45,044 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2016-12-24 05:28:45,060 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.datanode is not defined
2016-12-24 05:28:45,081 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2016-12-24 05:28:45,085 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
2016-12-24 05:28:45,092 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2016-12-24 05:28:45,092 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2016-12-24 05:28:45,144 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 33633
2016-12-24 05:28:45,144 INFO org.mortbay.log: jetty-6.1.26
2016-12-24 05:28:45,533 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33633
2016-12-24 05:28:45,780 INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:50075
2016-12-24 05:28:46,441 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = root
2016-12-24 05:28:46,447 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup
2016-12-24 05:28:46,638 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2016-12-24 05:28:46,729 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2016-12-24 05:28:46,771 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
2016-12-24 05:28:46,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
2016-12-24 05:28:46,827 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
2016-12-24 05:28:46,846 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to Hadoop/192.168.88.137:54310 starting to offer service
2016-12-24 05:28:46,868 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2016-12-24 05:28:46,870 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2016-12-24 05:28:47,768 INFO org.apache.hadoop.hdfs.server.common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1)
2016-12-24 05:28:47,780 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /var/lib/hadoop/hdfs/datanode/in_use.lock acquired by nodename 6952@ubuntu
2016-12-24 05:28:47,788 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/var/lib/hadoop/hdfs/datanode/
java.io.IOException: Incompatible clusterIDs in /var/lib/hadoop/hdfs/datanode: namenode clusterID = CID-558e02e9-5f72-47a7-a165-b931abbab42c; datanode clusterID = CID-9ce648f5-4684-4895-8cda-260b845a29e8
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:775)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:300)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:416)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:395)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:573)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1362)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1327)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
    at java.lang.Thread.run(Thread.java:745)
2016-12-24 05:28:47,804 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to Hadoop/192.168.88.137:54310. Exiting. 
java.io.IOException: All specified directories are failed to load.
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:574)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1362)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1327)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
    at java.lang.Thread.run(Thread.java:745)
2016-12-24 05:28:47,804 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to Hadoop/192.168.88.137:54310
2016-12-24 05:28:47,810 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
2016-12-24 05:28:49,811 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2016-12-24 05:28:49,812 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2016-12-24 05:28:49,814 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at ubuntu/127.0.1.1
************************************************************/

Now it works well I only leave these lines in my hdfs-site.xml

<property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>

Is this going to pose problems ?

On the graphical interface of my cluster I see only a single datanode you can see datanodes information her enter image description here

Thnak you

hadoop hadoop-streaming
Original Q&A
1

There are 1 answers

4
code On BEST ANSWER

Property fs.default.name is deprecated. Try below instead.

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://NAME_NODE_HOST:8020</value>
</property>

After looking at the logs:

Looks like you need to format the namenode.

Please try iceberg's solution at Datanode not starts correctly

Related Questions in HADOOP

Related Questions in HADOOP-STREAMING

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions