We have a spark 2 HDInsight cluster which has 650 GB and 195 Vcores. This is a 9 worker nodes and 2 head nodes cluster. The problem is that the jobs are not fully utilizing the cluster. For example when I run one job, Its using only 164 GB of memory when we have 650 GB. I have solved this problem by increasing the spark.executor.memory to 40 GB from 10 GB. The spark.executor.instances is 16 . But again the problem comes when I run multiple jobs. The job which came first will use the entire cluster till it finishes. The other jobs will just be in running mode with only 3 GB of memory. The requirement is that , the cluster has to be fully utilized when only one job is running. If there are multiple jobs, the resources ( Ram and Vcores ) have to be shared among the jobs.
Dynamic allocation of spark 2 cluster resources to the running jobs
206 views Asked by TomG At
1
There are 1 answers
Related Questions in APACHE-SPARK
- Getting error while running spark-shell on my system; pyspark is running fine
- ingesting high volume small size files in azure databricks
- Spark load all partions at once
- Databricks Delta table / Compute job
- Autocomplete not working for apache spark in java vscode
- How to overwrite a single partition in Snowflake when using Spark connector
- Parse multiple record type fixedlength file with beanio gives oom and timeout error for 10GB data file
- includeExistingFiles: false does not work in Databricks Autoloader
- Spark connectors from Azure Databricks to Snowflake using AzureAD login
- SparkException: Task failed while writing rows, caused by Futures timed out
- Configuring Apache Spark's MemoryStream to simulate Kafka stream
- Databricks can't find a csv file inside a wheel I installed when running from a Databricks Notebook
- Add unique id to rows in batches in Pyspark dataframe
- Does Spark Dynamic Allocation depend on external shuffle service to work well?
- Does Spark structured streaming support chained flatMapGroupsWithState by different key?
Related Questions in HADOOP-YARN
- Get yarn cache dir error: Command failed: yarn config get enableGlobalCache
- Spark Driver vs MapReduce Driver on YARN
- How to set spark.executor.extraClassPath & spark.driver.extraClassPath in hive query without adding those in hive-site.xml
- Yarn berry can't find type in module
- resource manager and nodemanager connectivity issues
- Spark with Yarn failing with S3 ClassNotFound on non-S3 tasks
- New Angular Installation yields a "EISDIR: illegal operation on a directory, read" error when running yarn but not npm
- Hadoop MiniCluster Web UI
- How do i fix hadoop error when Username has a space in it?
- Why is my vercel keep making Webpack error when deploying?
- Committing the yarn.lock File with Specified Versions in package.json?
- Does CDH 6.3.2 yarn have Resource or node restrictions？
- CDH 6.3.2 YARN's queue has a lots of pending applications,but yarn queue resources are sufficient
- Importing modules with yarn dlx @yarnpkg/sdks vscode command in latest version of TypeScript does not resolve issue TS2307
- Trouble installing packages with Yarn due to SSL certificate error
Related Questions in AZURE-HDINSIGHT
- hdfs library will not load in an HDinsight jupyter notebook
- Installing python packages on HDInsight on-demand cluster via Azure DataFactory ADF's spark activity
- Is it possible to use Azure Schema registry with HDInsights?
- Where to mention a "container" in the storage to store logs when on-demand HDInsight cluster gets created using Azure Data Factory?
- HDInsight cluster creation: <account> is not a valid ARM resource id
- Problem installing hadoop-gremlin with janusgraph
- Connect Azure Hadoop HDInsight Cluster with Azure data Factory
- How do I properly specify the number of HDInsight Kafka workers and disallow public IP address in my Azure HDInsight Kafka Terraform script?
- Terraform: unable to deploy Azure HDInsight
- Python Spark application does not end properly in Azure HDInsight (ERROR RawSocketSender, java.net.SocketException: Broken pipe)
- how to change python version from 2.7 to 3.5 in hdinsight spark
- Pass parameters/arguments to HDInsight/Spark Activity in Azure Data Factory
- not able to access azure keyvault from azure HD insights using managed identity
- Files not getting saved in Azure blob using Spark in HDInsights cluster
- how can i increase the core quota limit on microsoft.HDInsight azure?
Related Questions in APACHE-SPARK-2.0
- Spark SQL query fails with NullPointerException
- Parquet Size Difference
- Error importing Spark version 2.12.10 in Scala project Intellij IDE
- Unable to create spark pods in OCP with read write permission
- Spark-Hive Application: SASL Negotiation Failure with Kerberos on a Cluster
- Spark worker is killing the application
- Get day of week in Pyspark 3 using date format
- Maximum JDK version supported for apache spark 2.4.5.1
- How to free up Spark worker container memory after jobs finish
- Spark checkpointing data cleaning frequency
- Join 2 large size tables (50 Gb and 1 billion records)
- how to read CSV file with apache-spark spring boot
- Two big files join as one to many relationship in Java Spark
- How to configure pyspark to access AWS S3 containers?
- Writing parquet output with selected attributes from Bean
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
I suggest that you change your yarn scheduler to Capacity scheduler. This is better at sharing resources. This will help you to ensure the resources are better shared. By default hadoop is 'First in First out'. I respectfully disagree with 100% utilization of HDInsight. Your fix of increase executors to 40GB is exactly why a new job of 3GB can't get into your cluster. (And just because you allocate 40GB does not mean your job will use it.) If you want to increase cluster usage you might consider adding executors with less threads to fully utilize the cluster. This in combination with Capacity Scheduler & Pre-emption might be the answer to getting more performance and flexibility.