I am new to spark, I use the following code to define spark object:

         SparkSession spark = SparkSession
          .builder()
          .appName("Java Spark basic example")
          .config("spark.master", "local")
          .getOrCreate();

I get this exception:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.<init>(Lorg/apache/spark/sql/SparkSession;)V
at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:159)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:957)
at Spark.main(Spark.java:26)

It refers to line 26 where I wrote : .config("spark.master", "local").

Could you please explain why I am getting this error and how to correct it?

1 Answers

-1
Rajat Mishra On

Check the JAVA API docs. Below snippet of code should work :

import org.apache.spark.sql.SparkSession
SparkSession spark =   SparkSession.builder()
         .master("local")
         .appName("App Name")
         .config("spark.some.config.option", "some-value").
         .getOrCreate()