spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Attila Zsolt Piros <piros.attila.zs...@gmail.com>
Subject Re: Spark with External Shuffle Service - using saved shuffle files in the event of executor failure
Date Wed, 12 May 2021 16:38:46 GMT
Hello,

I have answered it on the Stack Overflow.

Best Regards,
Attila


On Wed, May 12, 2021 at 4:57 PM Chris Thomas <chris@heath-studios.com>
wrote:

> Hi,
>
> I am pretty confident I have observed Spark configured with the Shuffle
> Service continuing to fetch shuffle files on a node in the event of
> executor failure, rather than recompute the shuffle files as happens
> without the Shuffle Service. Can anyone confirm this?
>
> (I have a SO question
> <https://stackoverflow.com/questions/67466878/can-spark-with-external-shuffle-service-use-saved-shuffle-files-in-the-event-of>open
> on the same if you would rather answer directly there).
>
> Kind regards,
>
> Chris
>
>
>

Mime
View raw message