spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chris <christian.b...@performance-media.de>
Subject Re: Shuffle write increases in spark 1.2
Date Tue, 10 Feb 2015 09:35:49 GMT
Hello,

as the original message never got accepted to the mailinglist, I quote it
here completely:


Kevin Jung wrote
> Hi all,
> The size of shuffle write showing in spark web UI is much different when I
> execute same spark job on same input data(100GB) in both spark 1.1 and
> spark 1.2.
> At the same sortBy stage, the size of shuffle write is 39.7GB in spark 1.1
> but 91.0GB in spark 1.2.
> I set spark.shuffle.manager option to hash because it's default value is
> changed but spark 1.2 writes larger file than spark 1.1.
> Can anyone tell me why this happens?
> 
> Thanks
> Kevin

I'm experiencing the same thing with my job and that's what I tested:

* Spark 1.2.0 with Sort-based Shuffle
* Spark 1.2.0 with Hash-based Shuffle
* Spark 1.2.1 with Sort-based Shuffle

All three combinations show the same behaviour, which contrasts from Spark
1.1.0.

In Spark 1.1.0, my job runs for about an hour, in Spark 1.2.x it runs for
almost four hours. Configuration is identical otherwise - I only added
org.apache.spark.scheduler.CompressedMapStatus to the Kryo registrator for
Spark 1.2.0 to cope with https://issues.apache.org/jira/browse/SPARK-5102.


As a consequence (I think, but causality might be different) I see lots and
lots of disk spills.

I cannot provide a small test case, but maybe the log entries for a single
worker thread can help someone investigate on this. (See below.)


I will also open up an issue, if nobody stops me by providing an answer ;)

Any help will be greatly appreciated, because otherwise I'm stuck with Spark
1.1.0, as quadrupling runtime is not an option.

Sincerely,

Chris



2015-02-09T14:06:06.328+01:00	INFO	org.apache.spark.executor.Executor
Running task 9.0 in stage 18.0 (TID 300)	Executor task launch worker-18
2015-02-09T14:06:06.351+01:00	INFO	org.apache.spark.CacheManager	Partition
rdd_35_9 not found, computing it	Executor task launch worker-18
2015-02-09T14:06:06.351+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Getting 10 non-empty
blocks out of 10 blocks	Executor task launch worker-18
2015-02-09T14:06:06.351+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Started 0 remote
fetches in 0 ms	Executor task launch worker-18
2015-02-09T14:06:07.396+01:00	INFO	org.apache.spark.storage.MemoryStore
ensureFreeSpace(2582904) called with curMem=300174944, maxMe...	Executor
task launch worker-18
2015-02-09T14:06:07.397+01:00	INFO	org.apache.spark.storage.MemoryStore
Block rdd_35_9 stored as bytes in memory (estimated size 2.5...	Executor
task launch worker-18
2015-02-09T14:06:07.398+01:00	INFO
org.apache.spark.storage.BlockManagerMaster	Updated info of block rdd_35_9
Executor task launch worker-18
2015-02-09T14:06:07.399+01:00	INFO	org.apache.spark.CacheManager	Partition
rdd_38_9 not found, computing it	Executor task launch worker-18
2015-02-09T14:06:07.399+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Getting 10 non-empty
blocks out of 10 blocks	Executor task launch worker-18
2015-02-09T14:06:07.400+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Started 0 remote
fetches in 0 ms	Executor task launch worker-18
2015-02-09T14:06:07.567+01:00	INFO	org.apache.spark.storage.MemoryStore
ensureFreeSpace(944848) called with curMem=302757848, maxMem...	Executor
task launch worker-18
2015-02-09T14:06:07.568+01:00	INFO	org.apache.spark.storage.MemoryStore
Block rdd_38_9 stored as values in memory (estimated size 92...	Executor
task launch worker-18
2015-02-09T14:06:07.569+01:00	INFO
org.apache.spark.storage.BlockManagerMaster	Updated info of block rdd_38_9
Executor task launch worker-18
2015-02-09T14:06:07.573+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Getting 34 non-empty
blocks out of 50 blocks	Executor task launch worker-18
2015-02-09T14:06:07.573+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Started 0 remote
fetches in 1 ms	Executor task launch worker-18
2015-02-09T14:06:38.931+01:00	INFO	org.apache.spark.CacheManager	Partition
rdd_41_9 not found, computing it	Executor task launch worker-18
2015-02-09T14:06:38.931+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Getting 3 non-empty
blocks out of 10 blocks	Executor task launch worker-18
2015-02-09T14:06:38.931+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Started 0 remote
fetches in 0 ms	Executor task launch worker-18
2015-02-09T14:06:38.945+01:00	INFO	org.apache.spark.storage.MemoryStore
ensureFreeSpace(0) called with curMem=307529127, maxMem=9261...	Executor
task launch worker-18
2015-02-09T14:06:38.945+01:00	INFO	org.apache.spark.storage.MemoryStore
Block rdd_41_9 stored as bytes in memory (estimated size 0.0...	Executor
task launch worker-18
2015-02-09T14:06:38.946+01:00	INFO
org.apache.spark.storage.BlockManagerMaster	Updated info of block rdd_41_9
Executor task launch worker-18
2015-02-09T14:06:38.946+01:00	WARN	org.apache.spark.storage.BlockManager
Block rdd_41_9 replicated to only 0 peer(s) instead of 1 pee...	Executor
task launch worker-18
2015-02-09T14:06:39.088+01:00	INFO	org.apache.spark.storage.BlockManager
Found block rdd_3_9 locally	Executor task launch worker-18
2015-02-09T14:06:41.389+01:00	INFO	org.apache.spark.CacheManager	Partition
rdd_7_9 not found, computing it	Executor task launch worker-18
2015-02-09T14:06:41.389+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Getting 1 non-empty
blocks out of 1 blocks	Executor task launch worker-18
2015-02-09T14:06:41.389+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Started 0 remote
fetches in 0 ms	Executor task launch worker-18
2015-02-09T14:06:41.402+01:00	INFO	org.apache.spark.storage.MemoryStore
ensureFreeSpace(38144) called with curMem=307529151, maxMem=...	Executor
task launch worker-18
2015-02-09T14:06:41.402+01:00	INFO	org.apache.spark.storage.MemoryStore
Block rdd_7_9 stored as values in memory (estimated size 37....	Executor
task launch worker-18
2015-02-09T14:06:41.404+01:00	INFO
org.apache.spark.storage.BlockManagerMaster	Updated info of block rdd_7_9
Executor task launch worker-18
2015-02-09T14:07:00.019+01:00	INFO	org.apache.spark.CacheManager	Partition
rdd_73_9 not found, computing it	Executor task launch worker-18
2015-02-09T14:07:00.019+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Getting 1 non-empty
blocks out of 1 blocks	Executor task launch worker-18
2015-02-09T14:07:00.019+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Started 0 remote
fetches in 0 ms	Executor task launch worker-18
2015-02-09T14:07:00.185+01:00	INFO	org.apache.spark.storage.MemoryStore
ensureFreeSpace(826250) called with curMem=307567295, maxMem...	Executor
task launch worker-18
2015-02-09T14:07:00.185+01:00	INFO	org.apache.spark.storage.MemoryStore
Block rdd_73_9 stored as values in memory (estimated size 80...	Executor
task launch worker-18
2015-02-09T14:07:00.186+01:00	INFO
org.apache.spark.storage.BlockManagerMaster	Updated info of block rdd_73_9
Executor task launch worker-18
2015-02-09T14:07:00.190+01:00	INFO	org.apache.spark.storage.BlockManager
Found block rdd_35_9 locally	Executor task launch worker-18
2015-02-09T14:07:00.190+01:00	INFO	org.apache.spark.storage.BlockManager
Found block rdd_38_9 locally	Executor task launch worker-18
2015-02-09T14:07:00.194+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Getting 34 non-empty
blocks out of 50 blocks	Executor task launch worker-18
2015-02-09T14:07:00.194+01:00	INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator	Started 0 remote
fetches in 1 ms	Executor task launch worker-18
2015-02-09T14:07:17.967+01:00	INFO
org.apache.spark.util.collection.ExternalAppendOnlyMap	Thread 117 spilling
in-memory map of 670.2 MB to disk (1 tim...	Executor task launch worker-18
2015-02-09T14:07:46.716+01:00	INFO	org.apache.spark.storage.BlockManager
Found block rdd_41_9 locally	Executor task launch worker-18
2015-02-09T14:07:47.603+01:00	INFO	org.apache.spark.storage.BlockManager
Found block rdd_3_9 locally	Executor task launch worker-18
2015-02-09T14:07:47.897+01:00	INFO
org.apache.spark.util.collection.ExternalAppendOnlyMap	Thread 117 spilling
in-memory map of 5.0 MB to disk (1 time ...	Executor task launch worker-18
2015-02-09T14:07:48.270+01:00	INFO
org.apache.spark.util.collection.ExternalAppendOnlyMap	Thread 117 spilling
in-memory map of 5.0 MB to disk (2 times...	Executor task launch worker-18
2015-02-09T14:07:48.727+01:00	INFO
org.apache.spark.util.collection.ExternalAppendOnlyMap	Thread 117 spilling
in-memory map of 5.0 MB to disk (3 times...	Executor task launch worker-18
2015-02-09T14:07:49.021+01:00	INFO
org.apache.spark.util.collection.ExternalAppendOnlyMap	Thread 117 spilling
in-memory map of 5.0 MB to disk (4 times...	Executor task launch worker-18









--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-write-increases-in-spark-1-2-tp20894p21570.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message