I am trying to understand how the default configuration on cluster created in Dataproc is different from cluster created on EMR.
so i tried creating Single node cluster with Machine type 16vcpu 8 cores and 64 gb ram.
After creating cluster i see following properties
spark-env:SPARK_DAEMON_MEMORY
4000m
spark:spark.driver.maxResultSize
16384m
spark:spark.driver.memory
32768m
spark:spark.executor.cores
8
spark:spark.executor.instances
2
spark:spark.executor.memory
28604m
spark:spark.executorEnv.OPENBLAS_NUM_THREADS
1
spark:spark.jars.packages
io.delta:delta-core_2.12:2.3.0
spark:spark.scheduler.mode
FAIR
spark:spark.sql.cbo.enabled
true
spark:spark.ui.port
0
spark:spark.yarn.am.memory
640m
can anyone how driver memory and executor memory is getting calculated and on what factor cores are getting calculated and how this is diffrent from EMR cluster configurations. I am expecting if anyone can explain me calculations taking example machine type which i mentioned