I am having an issue on using PySpark with Google Cloud Dataproc. Currently, I have 2 similar clusters; one is autoscale and another is no-autoscale. When I declare Spark session with a specific app name on no-autoscale cluster, it seems fine to me both on notebook and Spark History Server with app name of modddddddd as I attached the picture below
However, on autoscale cluster, it doesn't work the same as previous cluster
And I looked up its app name on Spark History Server, it keeps saying PySparkShell instead of modddddddd.
I have tried to search for a solution but I found none. Is there anyway to solve this issue? Since this autoscale cluster has a lot of users, it's best for logging history on backend corresponding to individual app name.
Thank you in advance!