error of accessing first element in JavaPairRDD from Java SparkContext in IntelliJ

104 views Asked by At

I am trying to run an example Java/Spark code from IntelliJ Idea on MacBook pro.

My java:

  12.0.2 

Also, I have run:

  mvn dependency:copy-dependencies

in the folder with pom.xml.

It works well.

My code:

  SparkConf conf = new SparkConf()
                .setAppName("Word Count").setMaster("local");
  JavaSparkContext sc = new JavaSparkContext(conf);

  JavaDoubleRDD doubleRDD = sc.parallelizeDoubles(Arrays.asList(1.0, 1.0, 2.0, 3.0, 5.0, 8.0));
  JavaRDD<String> stringRDD = sc.parallelize(Arrays.asList("Hello", "World"));
  JavaPairRDD<String, Double> cartesian = stringRDD.cartesian(doubleRDD);
  cartesian.first(); // error!

I got error:

   java.lang.IllegalArgumentException

When I debugged into this, I found:

 the source code does not match the bytecode.

I am not sure why I got this. Did I do something wrong ? or I missed something to set up something for Spark ?

thanks

0

There are 0 answers