spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Melo <>
Subject Re: SPIP: Accelerator-aware Scheduling
Date Fri, 01 Mar 2019 17:19:13 GMT

On Fri, Mar 1, 2019 at 9:48 AM Xingbo Jiang <> wrote:
> Hi Sean,
> To support GPU scheduling with YARN cluster, we have to update the hadoop version to
3.1.2+. However, if we decide to not upgrade hadoop to beyond that version for Spark 3.0,
then we just have to disable/fallback the GPU scheduling with YARN, users shall still be able
to have that feature with Standalone or Kubernetes cluster.
> We didn't include the Mesos support in current SPIP because we didn't receive use cases
that require GPU scheduling on Mesos cluster, however, we can still add Mesos support in the
future if we observe valid use cases.

First time caller, long time listener. We have GPUs in our Mesos-based
Spark cluster, and it would be nice to use them with Spark-based
GPU-enabled frameworks (our use case is deep learning applications).


> Thanks!
> Xingbo
> Sean Owen <> 于2019年3月1日周五 下午10:39写道:
>> Two late breaking questions:
>> This basically requires Hadoop 3.1 for YARN support?
>> Mesos support is listed as a non goal but it already has support for requesting GPUs
in Spark. That would be 'harmonized' with this implementation even if it's not extended?
>> On Fri, Mar 1, 2019, 7:48 AM Xingbo Jiang <> wrote:
>>> I think we are aligned on the commitment, I'll start a vote thread for this shortly.
>>> Xiangrui Meng <> 于2019年2月27日周三 上午6:47写道:
>>>> In case there are issues visiting Google doc, I attached PDF files to the
>>>> On Tue, Feb 26, 2019 at 7:41 AM Xingbo Jiang <>
>>>>> Hi all,
>>>>> I want send a revised SPIP on implementing Accelerator(GPU)-aware Scheduling.
It improves Spark by making it aware of GPUs exposed by cluster managers, and hence Spark
can match GPU resources with user task requests properly. If you have scenarios that need
to run workloads(DL/ML/Signal Processing etc.) on Spark cluster with GPU nodes, please help
review and check how it fits into your use cases. Your feedback would be greatly appreciated!
>>>>> # Links to SPIP and Product doc:
>>>>> * Jira issue for the SPIP:
>>>>> * Google Doc:
>>>>> * Product Doc:
>>>>> Thank you!
>>>>> Xingbo

To unsubscribe e-mail:

View raw message