Supervisor node will not connect to storm cluster

823 views Asked by At

I have a three node cluster (2 supervisors, 1 nimbus) and am trying to add a third supervisor but it is not connecting to the cluster. I'm not quite sure why and am wondering anyone can take a look at my log file to see if they understand it more than me, seeing as I'm pretty new to computer science and the general field of computers.

    2015-06-09T12:02:56.674-0400 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, bui$
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=M7
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.6.0_35
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Sun Microsystems Inc.
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-6-openjd$
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/home/scix3/apache/s$
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client     environment:java.library.path=/usr/local/lib:/op$
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-45-generic
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=scix3
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/scix3
    2015-06-09T12:02:56.676-0400 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/home/scix3/apache/storm/lo$
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569$
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=M7
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.6.0_35
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Sun Microsystems$
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server     environment:java.home=/usr/lib/jvm/java-$
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/home/scix3/$
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local$
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.13.0-45-generic
    2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=scix3
     2015-06-09T12:02:56.684-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/scix3
    2015-06-09T12:02:56.685-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/home/scix3/apache/$
    2015-06-09T12:02:56.958-0400 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path" "/tm$
    2015-06-09T12:02:56.975-0400 b.s.u.StormBoundedExponentialBackoffRetry  [INFO] The baseSleepTimeMs [1000] the $
    2015-06-09T12:02:57.015-0400 o.a.s.c.f.i.CuratorFrameworkImpl [INFO]  Starting
    2015-06-09T12:02:57.017-0400 o.a.s.z.ZooKeeper [INFO] Initiating client  connection, connectString=192.168.2.1$
    2015-06-09T12:02:57.049-0400 o.a.s.c.f.i.CuratorFrameworkImpl [ERROR]      Background exception was not retry-able$
    java.net.UnknownHostException: 192.168.2.106"
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~ [na:1.6.0_35]
        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:902) ~[na:1.6.0_35]
        at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1281) ~[na:1.6.0_35]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1232) ~[na:1.6.0_35]
        at java.net.InetAddress.getAllByName(InetAddress.java:1163) ~[na:1.6.0_35]
        at java.net.InetAddress.getAllByName(InetAddress.java:1099) ~[na:1.6.0_35]
        at org.apache.storm.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) ~[storm-co$
        at org.apache.storm.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445) ~[storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:2$
        at org.apache.storm.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.j$
        at org.apache.storm.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94) ~[storm-core-0.9.4.jar:$
        at org.apache.storm.curator.HandleHolder.getZooKeeper(HandleHolder.java:55) ~[storm-core-0.9.4.jar:0.$
        at org.apache.storm.curator.ConnectionState.reset(ConnectionState.java:219) ~[storm-core-0.9.4.jar:0.$
        at org.apache.storm.curator.ConnectionState.start(ConnectionState.java:103) ~[storm-core-0.9.4.jar:0.$
        at org.apache.storm.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:190) ~[storm-cor$
        at org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:264) $
        at backtype.storm.zookeeper$mk_client.doInvoke(zookeeper.clj:79) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.RestFn.invoke(RestFn.java:494) ~[clojure-1.5.1.jar:na]
        at backtype.storm.cluster$mk_distributed_cluster_state.invoke(cluster.clj:42) ~[storm-core-0.9.4.jar:$
        at backtype.storm.cluster$mk_storm_cluster_state.invoke(cluster.clj:238) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.supervisor$supervisor_data.invoke(supervisor.clj:214) ~[storm-core-0.9.4.jar$
        at backtype.storm.daemon.supervisor$fn__5518$exec_fn__1754__auto____5519.invoke(supervisor.clj:409) ~$
        at clojure.lang.AFn.applyToHelper(AFn.java:167) ~[clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
        at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
        at  backtype.storm.daemon.supervisor$fn__5518$mk_supervisor__5544.doInvoke(superviso r.clj:405) ~[storm$
        at clojure.lang.RestFn.invoke(RestFn.java:436) ~[clojure-1.5.1.jar:na]
        at  backtype.storm.daemon.supervisor$fn__5518$mk_supervisor__5544.doInvoke(superviso r.clj:405) ~[storm$
        at clojure.lang.RestFn.invoke(RestFn.java:436) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor$_launch.invoke(supervisor.clj:629) ~  [storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.supervisor$_main.invoke(supervisor.clj:659) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.applyToHelper(AFn.java:159) ~[clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor.main(Unknown Source) ~[storm-core-0.9.4.jar:0.9.4]
    2015-06-09T12:03:12.059-0400 o.a.s.c.ConnectionState [ERROR] Connection timed out for connection string (192.$
    org.apache.storm.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
        at   org.apache.storm.curator.ConnectionState.checkTimeouts(ConnectionState.java:198) [storm-core-0.9.4$
        at org.apache.storm.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [storm-core-0.9.4.j$
        at org.apache.storm.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) ~[st$
        at org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.jav$
        at org.apache.storm.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172) ~[sto$
        at org.apache.storm.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161) ~[sto$
        at org.apache.storm.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:$
        at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148) ~[st$
        at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36) ~[sto$
        at backtype.storm.zookeeper$exists_node_QMARK_$fn__807.invoke(zookeeper.clj:101) ~[storm-core-0.9.4.j$
        at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:98) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:114) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.cluster$mk_distributed_cluster_state.invoke(cluster.clj:43) ~[storm-core-0.9.4.jar:$
        at backtype.storm.cluster$mk_storm_cluster_state.invoke(cluster.clj:238)  ~[storm-core-0.9.4.jar:0.9.4]
        at  backtype.storm.daemon.supervisor$supervisor_data.invoke(supervisor.clj:214) ~[storm-core-0.9.4.jar$
        at  backtype.storm.daemon.supervisor$fn__5518$exec_fn__1754__auto____5519.invoke(supervisor.clj:409) ~$
        at clojure.lang.AFn.applyToHelper(AFn.java:167) ~[clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
        at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor$fn__5518$mk_supervisor__5544.doInvoke(supervisor.clj:405) ~[storm$
        at clojure.lang.RestFn.invoke(RestFn.java:436) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor$_launch.invoke(supervisor.clj:629) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.supervisor$_main.invoke(supervisor.clj:659) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.applyToHelper(AFn.java:159) ~[clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor.main(Unknown Source) ~[storm-core-0.9.4.jar:0.9.4]
    2015-06-09T12:03:28.063-0400 o.a.s.c.ConnectionState [WARN] Connection attempt  unsuccessful after 31047 (grea$
    2015-06-09T12:03:28.063-0400 o.a.s.z.ZooKeeper [INFO] Initiating client  connection, connectString=192.168.2.1$
    2015-06-09T12:03:28.094-0400 b.s.d.supervisor [ERROR] Error on   initialization of server mk-supervisor
    java.lang.RuntimeException: java.net.UnknownHostException: 192.168.2.106"
        at backtype.storm.util$wrap_in_runtime.invoke(util.clj:44) ~[storm-core- 0.9.4.jar:0.9.4]
        at  backtype.storm.zookeeper$exists_node_QMARK_$fn__807.invoke(zookeeper.clj:102) ~ [storm-core-0.9.4.j$
        at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:98)  ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:114) ~[storm-core-0.9.4.jar:0.9.4]
        at  backtype.storm.cluster$mk_distributed_cluster_state.invoke(cluster.clj:43) ~ [storm-core-0.9.4.jar:$
        at backtype.storm.cluster$mk_storm_cluster_state.invoke(cluster.clj:238)  ~[storm-core-0.9.4.jar:0.9.4]
        at  backtype.storm.daemon.supervisor$supervisor_data.invoke(supervisor.clj:214) ~ [storm-core-0.9.4.jar$
        at  backtype.storm.daemon.supervisor$fn__5518$exec_fn__1754__auto____5519.invoke(sup ervisor.clj:409) ~$
        at clojure.lang.AFn.applyToHelper(AFn.java:167) ~[clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
        at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
        at  backtype.storm.daemon.supervisor$fn__5518$mk_supervisor__5544.doInvoke(superviso r.clj:405) ~[storm$
        at clojure.lang.RestFn.invoke(RestFn.java:436) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor$_launch.invoke(supervisor.clj:629) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.supervisor$_main.invoke(supervisor.clj:659) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.applyToHelper(AFn.java:159) ~[clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor.main(Unknown Source) ~[storm-core-0.9.4.jar:0.9.4]
Caused by: java.net.UnknownHostException: 192.168.2.106"
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.6.0_35]
        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:902) ~[na:1.6.0_35]
        at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1281) ~[na:1.6.0_35]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1232) ~[na:1.6.0_35]
        at java.net.InetAddress.getAllByName(InetAddress.java:1163) ~[na:1.6.0_35]
    at java.net.InetAddress.getAllByName0(InetAddress.java:1232) ~[na:1.6.0_35]
    at java.net.InetAddress.getAllByName(InetAddress.java:1163) ~[na:1.6.0_35]
    at java.net.InetAddress.getAllByName(InetAddress.java:1099) ~[na:1.6.0_35]
    at org.apache.storm.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) ~[storm-co$
    at org.apache.storm.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445) ~[storm-core-0.9.4.jar:0.9.4]
    at org.apache.storm.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:2$
    at org.apache.storm.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.j$
    at org.apache.storm.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94) ~[storm-core-0.9.4.jar:$
    at org.apache.storm.curator.HandleHolder.internalClose(HandleHolder.java:128) ~[storm-core-0.9.4.jar:$
    at org.apache.storm.curator.HandleHolder.closeAndReset(HandleHolder.java:77) ~[storm-core-0.9.4.jar:0$
    at org.apache.storm.curator.ConnectionState.reset(ConnectionState.java:218) ~[storm-core-0.9.4.jar:0.$
    at org.apache.storm.curator.ConnectionState.checkTimeouts(ConnectionState.java:194) ~[storm-core-0.9.$
    at org.apache.storm.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) ~[storm-core-0.9.4.$
    at org.apache.storm.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) ~[st$
    at org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.jav$
    at org.apache.storm.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172) ~[sto$
    at org.apache.storm.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161) ~[sto$
    at org.apache.storm.curator.RetryLoop.callWithRetry(RetryLoop.java:107) 

~[storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:$
        at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148) ~[st$
        at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36) ~[sto$
        at backtype.storm.zookeeper$exists_node_QMARK_$fn__807.invoke(zookeeper.clj:101) ~[storm-core-0.9.4.j$
        ... 16 common frames omitted
2015-06-09T12:03:28.103-0400 b.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
        at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) ~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.supervisor$fn__5518$mk_supervisor__5544.doInvoke(supervisor.clj:405) ~[storm$
at clojure.lang.RestFn.invoke(RestFn.java:436) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.supervisor$_launch.invoke(supervisor.clj:629) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.daemon.supervisor$_main.invoke(supervisor.clj:659) ~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.applyToHelper(AFn.java:159) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.supervisor.main(Unknown Source) ~[storm-core-0.9.4.jar:0.9.4]

Can anyone tell why my supervisor is not connecting from the given log file?

1

There are 1 answers

0
Max On

I screwed up. I was missing a quotation mark in my config file, which is why i got the "unknown host" error.