Installing splice machine on HDP 2.6.5

Question

We tried almost all the guides that we can find online to splice machine as a ambari service.

But everytime we run the sqlshell.sh, it just says there is no server running and unable to connect to port 1527 on localhost.

We have a simple HDP sandbox version 2.6.5 and a three node 2.6.5 cluster. We are trying to install version 2.87 of splicemachine.

These are the guides we followed.

https://github.com/splicemachine/spliceengine/blob/branch-2.8/platforms/hdp2.6.5/docs/HDP-installation.md

That did not work on our three node cluster

Then we tried the sandbox with this tutorial

https://github.com/splicemachine/splice-ambari-service

Again the same result.

Please let us know if there is anything that we have missed in the guide/ or are there any extra steps.

Answer 1

Can you confirm you are running sqlshell.sh on a server that is running a region server? I would look at the region server log files. You want to look for 'Ready to accept JDBC connections on 0.0.0.0:1527'. That is the indicator that a region server is up and running. If you do not see that, then can you look in the log file and see if there are any error messages?

Answer 2

Found out the answer - Apart from what is mentioned in the doc here - https://github.com/splicemachine/spliceengine/blob/branch-2.8/platforms/hdp2.6.5/docs/HDP-installation.md.

There were multiple NoClassDefFoundError errors being thrown in the hbase master logs for classes found in spark2 jars.

One of them was

    master.HMaster: The coprocessor com.splicemachine.hbase.SpliceMasterObserver threw java.io.IOException: Unexpected exception
java.io.IOException: Unexpected exception
        at com.splicemachine.si.data.hbase.coprocessor.CoprocessorUtils.getIOException(CoprocessorUtils.java:30)
        at com.splicemachine.hbase.SpliceMasterObserver.start(SpliceMasterObserver.java:111)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:415)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:256)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:159)
        at org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:93)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:774)
        at org.apache.hadoop.hbase.master.HMaster.access$900(HMaster.java:225)
        at org.apache.hadoop.hbase.master.HMaster$3.run(HMaster.java:2038)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/spark_project/guava/util/concurrent/ThreadFactoryBuilder
        at com.splicemachine.timestamp.impl.TimestampClient.<init>(TimestampClient.java:108)
        at com.splicemachine.timestamp.hbase.ZkTimestampSource.initialize(ZkTimestampSource.java:62)
        at com.splicemachine.timestamp.hbase.ZkTimestampSource.<init>(ZkTimestampSource.java:48)
        at com.splicemachine.si.data.hbase.coprocessor.HBaseSIEnvironment.<init>(HBaseSIEnvironment.java:146)
        at com.splicemachine.si.data.hbase.coprocessor.HBaseSIEnvironment.loadEnvironment(HBaseSIEnvironment.java:100)
        at com.splicemachine.hbase.SpliceMasterObserver.start(SpliceMasterObserver.java:81)
        ... 8 more
Caused by: java.lang.ClassNotFoundException: org.spark_project.guava.util.concurrent.ThreadFactoryBuilder
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 14 more

We found out that this class was in the spark-network-common jar and symlinked it to one of the folders that was part of HBASE_CLASSPATH. But then it threw another classdef not found error for a different class.

We ended up symlinking all the jars in spark2/jars folder to one of the folders that was part of HBASE_CLASSPATH.

After this the hbase master came up successfully and started the splice db process and we were able to connect with the sqlshell.sh

Nots: Make sure to do this symlinking on all nodes that have a regionserver.

What helped us in figuring this out were these two java files in splice documentation

https://github.com/splicemachine/spliceengine/blob/master/hbase_sql/src/main/java/com/splicemachine/hbase/SpliceMasterObserver.java

and

https://github.com/splicemachine/spliceengine/blob/ee61cadf17c97a0c5d866e2b764142f8e55311a5/splice_timestamp_api/src/main/java/com/splicemachine/timestamp/impl/TimestampServer.java