So I'm trying to get a simple example program working using Java and spark cassandra connector. Running sbt assembly works fine and I get a fat jar that I submit to spark. The problem comes here, when I submit the job to spark I get this error:
vagrant@cassandra-spark:~$ source submit-job.sh
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/spark/connector/japi/CassandraJavaUtil
at JavaTest.main(JavaTest.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.datastax.spark.connector.japi.CassandraJavaUtil
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
This is the submit-job.sh script below:
#!/usr/bin/env bash
~/spark/bin/spark-submit --driver-class-path ~/JavaTest/lib/spark-cassandra-connector-assembly-1.3.0-M2-SNAPSHOT.jar ~/JavaTest/target/scala-2.10/CassSparkTest-assembly-1.0.jar
This is my build.sbt file
lazy val root = (project in file(".")).
settings(
name := "CassSparkTest",
version := "1.0"
)
libraryDependencies ++= Seq(
"com.datastax.cassandra" % "cassandra-driver-core" % "2.1.5" % "provided",
"org.apache.cassandra" % "cassandra-thrift" % "2.1.5" % "provided",
"org.apache.cassandra" % "cassandra-clientutil" % "2.1.5" % "provided",
//"com.datastax.spark" %% "spark-cassandra-connector" % "1.3.0-M1" % "provided",
"com.datastax.spark" %% "spark-cassandra-connector-java" % "1.3.0-M1" % "provided",
"org.apache.spark" %% "spark-core" % "1.3.1" % "provided",
"org.apache.spark" %% "spark-streaming" % "1.3.1" % "provided",
"org.apache.spark" %% "spark-sql" % "1.3.1" % "provided",
"org.apache.commons" % "commons-lang3" % "3.4" % "provided"
)
Below is the code being compiled:
import static com.datastax.spark.connector.japi.CassandraJavaUtil.*;
import com.datastax.spark.connector.japi.CassandraRow;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import org.apache.commons.lang3.StringUtils;
public class JavaTest {
public static void main(String[] args) {
SparkConf conf = new SparkConf().set("spark.cassandra.connection.host", "127.0.0.1");
JavaSparkContext sc = new JavaSparkContext("spark://192.168.10.11:7077", "test", conf);
JavaRDD<String> cassandraRowsRDD = javaFunctions(sc).cassandraTable("ks", "test")
.map(new Function<CassandraRow, String>() {
@Override
public String call(CassandraRow cassandraRow) throws Exception {
return cassandraRow.toString();
}
});
System.out.println("Data as CassandraRows: \n" + StringUtils.join(cassandraRowsRDD.toArray(), "\n"));
}
}
sbt assembly works fine but the class definition isn't found once the job is actually submitted.
When I had hit this before I had to add the current class jar to the SparkConf. For example: