spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingjie Tang (JIRA)" <>
Subject [jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark
Date Thu, 02 Aug 2018 19:31:00 GMT


Mingjie Tang commented on SPARK-24615:

>From user's perspective, user only concern about the GPU resource for RDD, and do not
understand the stage or partitions of RDD. Therefore, underline resource allocation mechanism
would  assign the resources to executor automatically. 

Similar as cache or persistence to different level, maybe we can provide different configuration
to users. Then, resource allocation to follow the predefined policy to allocate resource. 

> Accelerator-aware task scheduling for Spark
> -------------------------------------------
>                 Key: SPARK-24615
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.4.0
>            Reporter: Saisai Shao
>            Assignee: Saisai Shao
>            Priority: Major
>              Labels: Hydrogen, SPIP
> In the machine learning area, accelerator card (GPU, FPGA, TPU) is predominant compared
to CPUs. To make the current Spark architecture to work with accelerator cards, Spark itself
should understand the existence of accelerators and know how to schedule task onto the executors
where accelerators are equipped.
> Current Spark’s scheduler schedules tasks based on the locality of the data plus the
available of CPUs. This will introduce some problems when scheduling tasks with accelerators
>  # CPU cores are usually more than accelerators on one node, using CPU cores to schedule
accelerator required tasks will introduce the mismatch.
>  # In one cluster, we always assume that CPU is equipped in each node, but this is not
true of accelerator cards.
>  # The existence of heterogeneous tasks (accelerator required or not) requires scheduler
to schedule tasks with a smart way.
> So here propose to improve the current scheduler to support heterogeneous tasks (accelerator
requires or not). This can be part of the work of Project hydrogen.
> Details is attached in google doc. It doesn't cover all the implementation details, just
highlight the parts should be changed.
> CC [~yanboliang] [~merlintang]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message