spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabrizio Milo aka misto <>
Subject Local Standalone Application and shuffle spills
Date Thu, 13 Mar 2014 01:22:09 GMT
Hello everyone

I have a question about Shuffle Spills. From the introduction to
amplab spark internals
each task output could be saved to disk for 'redundancy'

if I set spark.shuffle.spill to false would this behavior be
eliminated and make it in a way that it will never spill to disk ?

Thank you

Twitter: @fabmilo
Simplicity, consistency, and repetition - that's how you get through.
(Jack Welch)
Perfection must be reached by degrees; she requires the slow hand of
time (Voltaire)
The best way to predict the future is to invent it (Alan Kay)

View raw message