Not able to connect to local spark session

Question

Spark novice here. According to the instructions i found on web, I installed spark on my local windows development machine. The Spark properties from WebUI are

spark.app.name  Spark shell
spark.driver.host   192.168.2.131
spark.driver.port   53796
spark.executor.id   driver
spark.home  C:\BigData\spark
spark.master    local[*]

Now in my java web application in my local machine,when I am trying to create connect to the spark session using the code, a new local spark session is created which points to a different host and port even being in the same machine.

    SparkConf conf = new SparkConf()
                        .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
    Globals.spark = SparkSession.builder()
                            .master("local[2]")
                            .appName("test")
                            .config(conf)
                            .getOrCreate();     

Properties of new created spark session

spark.app.name  test
spark.driver.host   192.168.2.198
spark.driver.port   57755
spark.executor.id   driver
spark.master    local[2]

How do I connect to the spark session already running in my local machine? I want to do this because i want to be able to add executors and modify properties in my local machine to debug the application before getting it to production cluster.

Answer 1

This is not how Spark works: once it's running, you cannot connect externally to a Spark session. If you want to tune it, modify some parameters, you have to do it before.

Look at https://developer.ibm.com/code/open/projects/spark-bench/. Spark Bench will allow you to do that in an automated way. I hope this will help you solve your use-case.