spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Graves <>
Subject Re: YARN Shuffle service and its compatibility
Date Tue, 19 Apr 2016 14:44:59 GMT
It would be nice if we could keep this compatible between 1.6 and 2.0 so I'm more for Option
B at this point since the change made seems minor and we can change to have shuffle service
do internally like Marcelo mention. Then lets try to keep compatible, but if there is a forcing
function lets figure out a good way to run 2 at once.


    On Monday, April 18, 2016 5:23 PM, Marcelo Vanzin <> wrote:

 On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <> wrote:
> IIUC, the reason for that PR is that they found the string comparison to
> increase the size in large shuffles. Maybe we should add the ability to
> support the short name to Spark 1.6.2?

Is that something that really yields noticeable gains in performance?

If it is, it seems like it would be simple to allow executors register
with the full class name, and map the long names to short names in the
shuffle service itself.

You could even get fancy and have different ExecutorShuffleInfo
implementations for each shuffle service, with an abstract
"getBlockData" method that gets called instead of the current if/else


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message