spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes
Date Thu, 31 Oct 2019 20:00:03 GMT

    [ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964360#comment-16964360
] 

Thomas Graves edited comment on SPARK-29415 at 10/31/19 7:59 PM:
-----------------------------------------------------------------

More details:

TaskResourceRequest - this supports taking a resourceName and an amount (Double for fractional
resources).  It only supports cpus (spark.task.cpus) and accelerator resource types (spark.*.resource.[resourceName].*. 
So user can specify cpus and resources like GPU's and FPGAS. The accerator type resources
match what we already have for configs in the acceralator aware scheduling. [https://github.com/apache/spark/blob/master/docs/configuration.md#custom-resource-scheduling-and-configuration-overview]

 

ExecutorResourceRequest - this supports specifying the requirements for the executors.  It
supports all the configs needed for accelerator aware scheduling - {{spark.\{executor/driver}.resource.\{resourceName}.\{amount,
discoveryScript, vendor} as well as heap memory, overhead memory, pyspark memory, and cores.
In order to support memory types we added a "units" parameter into ExecutorResourceRequest. 
the other parameters resourceName, vendor, discoveryScript, amount all match the accelerator
aware scheduling parameters.}}

 

ResourceProfile - this class takes in the executor and task requirement and holds them to
be used by other components.  For instance, we have to pass the executor resources into the
cluster managers so it can ask for the proper containers.  The requests have to also be passed
into the executors when launched so they use the correct discovery Script.  The task requirements
are used by the scheduler to assign tasks to proper containers. {{  We also have a ResourceProfile
object that has an accessor to get the default ResourceProfile. This is the profile generated
from the configs the user passes in when the spark application is submitted.  So it will
have --executor-cores, memory, overhead memory, pyspark memory, accelerator resources the
user all specified via --confs or properties file on submit.}}


was (Author: tgraves):
More details:

TaskResourceRequest - this supports taking a resourceName and an amount (Double for fractional
resources).  It only supports cpus (spark.task.cpus) and accelerator resource types (spark.*.resource.[resourceName].*. 
So user can specify cpus and resources like GPU's and FPGAS. The accerator type resources
match what we already have for configs in the acceralator aware scheduling. [https://github.com/apache/spark/blob/master/docs/configuration.md#custom-resource-scheduling-and-configuration-overview]

 

ExecutorResourceRequest - this supports specifying the requirements for the executors.  It
supports all the configs needed for accelerator aware scheduling - {{spark.\{executor/driver}.resource.\{resourceName}.\{amount,
discoveryScript, vendor} as well as heap memory, overhead memory, pyspark memory, and cores.
In order to support memory types we added a "units" parameter into ExecutorResourceRequest. 
the other parameters resourceName, vendor, discoveryScript, amount all match the accelerator
aware scheduling parameters.}}

{{}}

ResourceProfile - this class takes in the executor and task requirement and holds them to
be used by other components.  For instance, we have to pass the executor resources into the
cluster managers so it can ask for the proper containers.  The requests have to also be passed
into the executors when launched so they use the correct discovery Script.  The task requirements
are used by the scheduler to assign tasks to proper containers. {{  We also have a ResourceProfile
object that has an accessor to get the default ResourceProfile. This is the profile generated
from the configs the user passes in when the spark application is submitted.  So it will
have --executor-cores, memory, overhead memory, pyspark memory, accelerator resources the
user all specified via --confs or properties file on submit.}}

> Stage Level Sched: Add base ResourceProfile and Request classes
> ---------------------------------------------------------------
>
>                 Key: SPARK-29415
>                 URL: https://issues.apache.org/jira/browse/SPARK-29415
>             Project: Spark
>          Issue Type: Story
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Major
>
> this is just to add initial ResourceProfile, ExecutorResourceRequest and taskResourceRequest
classes that are used by the other parts of the code.
> Initially we will have them private until we have other pieces in place.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message