spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 曹雪林 <>
Subject When will spark support "push" style shuffle?
Date Thu, 08 Jan 2015 01:47:15 GMT

      I've heard a lot of complain about spark's "pull" style shuffle. Is
there any plan to support "push" style shuffle in the near future?

      Currently, the shuffle phase must be completed before the next stage
starts. While, it is said, in Impala, the shuffled data is "streamed" to
the next stage handler, which greatly saves time. Will spark support this
mechanism one day?


View raw message