spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bijay Pathak <bijay.pat...@cloudwick.com>
Subject Shuffle Spill Memory and Shuffle Spill Disk
Date Mon, 23 Mar 2015 21:25:11 GMT
Hello,

I am running  TeraSort <https://github.com/ehiggs/spark-terasort> on 100GB
of data. The final metrics I am getting on Shuffle Spill are:

Shuffle Spill(Memory): 122.5 GB
Shuffle Spill(Disk): 3.4 GB

What's the difference and relation between these two metrics? Does these
mean 122.5 GB was spill from memory during the shuffle?

thank you,
bijay

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message