Cannot read cassandra column family on a spark cluster using the spark-cassandra-connector in java

Question

Here is my code to simply read a column family using the spark cassandra connector

import static com.datastax.spark.connector.japi.CassandraJavaUtil.*;

import com.datastax.spark.connector.japi.SparkContextJavaFunctions;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaRDD;

public class Main {
    private static final String HOST = "spark://sparkmaster:7077";
//    private static final String HOST = "local[4]";

    private static final String APP_NAME = "Cassandra Spark WordCount";

    public static void main (String... args) {
        String[] jars = {
                "./build/libs/CassandraSparkMapReduce-1.0-SNAPSHOT.jar"
        };

        SparkConf conf = new SparkConf(true)
                .set("spark.cassandra.connection.host", "107.108.214.154")
                .set("spark.executor.userClassPathFirst", "true")
                .setJars(jars);

        SparkContext sc = new SparkContext(HOST, APP_NAME, conf);
        SparkContextJavaFunctions context = javaFunctions(sc);

        JavaRDD<String> rdd = context.cassandraTable("wordcount", "input")
                .map(row -> row.toString());

        System.out.println(rdd.toArray());
    }
}

and here is my build.gradle file to build and run the application

group 'in.suyash.tests'
version '1.0-SNAPSHOT'

apply plugin: 'java'
apply plugin: 'application'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compile group: 'org.apache.spark', name: 'spark-core_2.10', version: '1.4.0'

    compile group: 'com.datastax.spark', name: 'spark-cassandra-connector_2.10', version: '1.4.0-M1'
    compile group: 'com.datastax.spark', name: 'spark-cassandra-connector-java_2.10', version: '1.4.0-M1'

    testCompile group: 'junit', name: 'junit', version: '4.11'
}

sourceSets {
    main {
        java {
            srcDir './'
        }
    }
}

mainClassName = 'Main'

// http://stackoverflow.com/a/14441628/3673043
jar {
    doFirst {
        from {
            configurations.compile.collect {
                it.isDirectory() ? it : zipTree(it)
            }
        }
    }
    exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
}

I execute my job by first doing gradle build to build the jar, and then I do gradle run. But the job fails, and looking at the stderr in my executors, I am getting the following exception

java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:194)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I have a 3 node setup, in which one node acts as the spark master node, while other two are spark worker nodes and also form a cassandra ring. I can execute the job locally, if I change my spark host, but on the cluster I am getting this weird exception, that I cannot find anywhere else. Versions:

• Spark: 1.4.0
• Cassandra: 2.1.6
• spark-cassandra-connector: 1.4.0-M1

EDIT

I can't exactly answer how I resolved this, but I removed all java installations from all my nodes, restarted everything and installed a fresh copy of jdk1.8.0_45, started my cluster again, and now the job completes successfully. Any explanations to this behaviour are welcome.