Unable to set config in spark-submit from command line

Question

I am trying to set master URL in the Application jar using below:

val spark = SparkSession
  .builder()
  .master("spark://master:7077")
  .appName("TestApp")
  .config("spark.sql.warehouse.dir", "/tmp/spark-warehouse")
  .getOrCreate()

I try to run the jar with below parameters:

spark-submit -—num-executors 10 -—executor-memory 5G appJar.jar 

spark-submit does not take the explict options provided to the jar. When the master URL is set in application jar, every other config is ignored? Can someone explain this please?

Answer 1

I think correct way is to give --

spark-submit --num-executors 10 --executor-memory 5G appJar.jar