spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bijay Pathak <bijay.pat...@cloudwick.com>
Subject Re: Shuffle Spill Memory and Shuffle Spill Disk
Date Mon, 23 Mar 2015 21:52:30 GMT
It looks this is not the right place for this question, I have send the
question to user group.

thank you,
bijay

On Mon, Mar 23, 2015 at 2:25 PM, Bijay Pathak <bijay.pathak@cloudwick.com>
wrote:

> Hello,
>
> I am running  TeraSort <https://github.com/ehiggs/spark-terasort> on
> 100GB of data. The final metrics I am getting on Shuffle Spill are:
>
> Shuffle Spill(Memory): 122.5 GB
> Shuffle Spill(Disk): 3.4 GB
>
> What's the difference and relation between these two metrics? Does these
> mean 122.5 GB was spill from memory during the shuffle?
>
> thank you,
> bijay
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message