spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuanjian Li <xyliyuanj...@gmail.com>
Subject Re: Design for continuous processing shuffle
Date Mon, 07 May 2018 14:15:26 GMT
Hi Joseph and devs,

Happy to see the discussion of CP shuffle, as comment in https://issues.apache.org/jira/browse/SPARK-20928?focusedCommentId=16245556&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16245556
<https://issues.apache.org/jira/browse/SPARK-20928?focusedCommentId=16245556&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16245556>,
the team also do some design and demo work over CP shuffle, the docs show more detailed work:

https://docs.google.com/document/d/14cGJ75v9myznywtB35ytEqL9wHy9xfZRv06B6g2tUgI <https://docs.google.com/document/d/14cGJ75v9myznywtB35ytEqL9wHy9xfZRv06B6g2tUgI>



> 在 2018年5月5日,02:27,Joseph Torres <joseph.torres@databricks.com> 写道:
> 
> Hi all,
> 
> A few of us have been working on a design for how to do shuffling in continuous processing.
Feel free to chip in if you have any comments or questions.
> 
> doc:
> https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE <https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE>
> 
> continuous processing SPIP: https://issues.apache.org/jira/browse/SPARK-20928 <https://issues.apache.org/jira/browse/SPARK-20928>
> 
> 
> Jose


Mime
View raw message