spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Imbalanced shuffle read
Date Wed, 12 Nov 2014 06:52:54 GMT
When you calls the groupByKey() try providing the number of partitions like
groupByKey(100) depending on your data/cluster size.

Thanks
Best Regards

On Wed, Nov 12, 2014 at 6:45 AM, ankits <ankitsoni9@gmail.com> wrote:

> Im running a job that uses groupByKey(), so it generates a lot of shuffle
> data. Then it processes this and writes files to HDFS in a forEachPartition
> block. Looking at the forEachPartition stage details in the web console,
> all
> but one executor is idle (SUCCESS in 50-60ms), and one is RUNNING with a
> huge shuffle read and takes a long time to finish.
>
> Can someone explain why the read is all on one node and how to parallelize
> this better?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Imbalanced-shuffle-read-tp18648.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message