spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mridul Muralidharan <mri...@gmail.com>
Subject Re: Push-based shuffle SPIP
Date Mon, 24 Aug 2020 21:53:47 GMT
Hi,

  Thanks for sending out the proposal Min !
For the SPIP requirements, I am willing to act as the shepherd for this
proposal.

The jira + paper + proposal provides the high level design and
implementation details.
The vldb paper discusses the performance gains in detail for the inhouse
deployment of push based shuffle.

Would be great to get feedback from our community on this feature; before
we go to voting.


Regards,
Mridul



On Mon, Aug 24, 2020 at 4:32 PM mshen <mshen@apache.org> wrote:

> We raised this SPIP ticket in
> https://issues.apache.org/jira/browse/SPARK-30602 earlier this year.
> Since then, we have progressed in multiple fronts, including:
>
> * Our work is published in VLDB 2020. The final version of the paper is
> attached in the SPIP ticket.
> * We have further enhanced and productionized this work at LinkedIn, and
> have enabled production flows adopting the new push-based shuffle
> mechanism,
> with good results.
> * We have recently also ported our push-based shuffle changes to OSS Spark
> master branch, so other people can potentially try it out. Details of this
> branch is in this  doc
> <
> https://docs.google.com/document/d/16yOfI8P_O3V6hx_FnWT22jeDIItgXuXfaDAV0fJDTqQ/edit#>
>
> * The  SPIP doc
> <
> https://docs.google.com/document/d/1mYzKVZllA5Flw8AtoX7JUcXBOnNIDADWRbJ7GI6Y71Q/edit>
>
> is also further updated reflecting more recent designs.
> * We have also discussed with multiple companies who share similar interest
> in this work.
>
> We would like to resume the discussion of this SPIP in the community, and
> push for a voting on this.
>
>
>
>
> -----
> Min Shen
> Staff Software Engineer
> LinkedIn
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message