spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave DeCaprio (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-26957) Add config properties to configure the default scheduler pool priorities
Date Thu, 21 Feb 2019 17:33:00 GMT
Dave DeCaprio created SPARK-26957:
-------------------------------------

             Summary: Add config properties to configure the default scheduler pool priorities
                 Key: SPARK-26957
                 URL: https://issues.apache.org/jira/browse/SPARK-26957
             Project: Spark
          Issue Type: Improvement
          Components: Scheduler
    Affects Versions: 2.4.0
            Reporter: Dave DeCaprio


Currently, it is possible to dynamically create new scheduler pools in Spark just by setting {{spark.scheduler.pool.}}
to a new value.

We use this capability to create separate pools for different projects that run jobs in the
same long-lived driver application. Each project gets its own pool, and within the pool jobs
are executed in a FIFO manner.

This setup works well, except that we also have a low priority queue for background tasks.
We would prefer for all of the dynamic pools to have a higher priority than this background
queue. 
 We can accomplish this by hardcoding the project queue names in a spark_pools.xml config
file and setting their priority to 100.

Unfortunately, there is no way to set the priority for dynamically created pools.  They
are all hardcoded to 1.  It would be nice if there were configuration settings to change
this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message