spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bijay Pathak <>
Subject Shuffle Spill Memory and Shuffle Spill Disk
Date Mon, 23 Mar 2015 21:29:50 GMT

I am running  TeraSort <> on 100GB
of data. The final metrics I am getting on Shuffle Spill are:

Shuffle Spill(Memory): 122.5 GB
Shuffle Spill(Disk): 3.4 GB

What's the difference and relation between these two metrics? Does these
mean 122.5 GB was spill from memory during the shuffle?

thank you,

View raw message