spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bijay Pathak <>
Subject Re: why "Shuffle Write" is not zero when everything is cached and there is enough memory?
Date Tue, 31 Mar 2015 20:19:37 GMT
The Spark Sort-Based Shuffle (default from 1.1) keeps the data from
each Map tasks to memory until they they can't fit after which they
are sorted and spilled to disk. You can reduce the Shuffle write to
disk by increasing spark.shuffle.memoryFraction(default 0.2).

By writing the shuffle output to disk the Spark lineage can be
truncated when the RDDs are already materialized as the side-effects
of earlier shuffle.This is the under the hood optimization in Spark
which is only possible because of shuffle output output being written
to disk.

You can set spark.shuffle.spill to false if you don't want to spill to
the disk and assuming you have enough heap memory.

On Tue, Mar 31, 2015 at 12:35 PM, Udit Mehta <> wrote:
> I have noticed a similar issue when using spark streaming. The spark shuffle
> write size increases to a large size(in GB) and then the app crashes saying:
> /data/vol0/nodemanager/usercache/$user/appcache/application_1427480955913_0339/spark-local-20150330231234-db1a/0b/temp_shuffle_1b23808f-f285-40b2-bec7-1c6790050d7f
> (No such file or directory)
> I dont understand why the shuffle size increases to such a large value for
> long running jobs.
> Thanks,
> Udiy
> On Mon, Mar 30, 2015 at 4:01 AM, shahab <> wrote:
>> Thanks Saisai. I will try your solution, but still i don't understand why
>> filesystem should be used where there is a plenty of memory available!
>> On Mon, Mar 30, 2015 at 11:22 AM, Saisai Shao <>
>> wrote:
>>> Shuffle write will finally spill the data into file system as a bunch of
>>> files. If you want to avoid disk write, you can mount a ramdisk and
>>> configure "spark.local.dir" to this ram disk. So shuffle output will write
>>> to memory based FS, and will not introduce disk IO.
>>> Thanks
>>> Jerry
>>> 2015-03-30 17:15 GMT+08:00 shahab <>:
>>>> Hi,
>>>> I was looking at SparkUI, Executors, and I noticed that I have 597 MB
>>>> for  "Shuffle while I am using cached temp-table and the Spark had 2 GB free
>>>> memory (the number under Memory Used is 597 MB /2.6 GB) ?!!!
>>>> Shouldn't be Shuffle Write be zero and everything (map/reduce) tasks be
>>>> done in memory?
>>>> best,
>>>> /Shahab

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message