spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Tuero <>
Subject Re: Spark 2.4 partitions and tasks
Date Mon, 25 Feb 2019 14:50:14 GMT
Good question.  What I have read about is that Spark is not a magician and
can't know how many tasks will be better for your input, so it can fail.
Spark set the default parallelism as twice the number of cores on the
In my jobs, it seemed that using the parallelism inherited from input parts
worked well sometimes, and it was 100x the default parallelism.
When every job started to use default parallelism (apparently when
switching from emr 5.16 to 5.20), I first tried to do some repartitions but
in some cases, it was the same: The repartition job took as long as the job
I wanted to affect (or failed directly).
Doing the repartition inside some operation on Rdd pairs worked really
better (https: //

It will be nice to have a more comprehensive look at which Rdds should need
more or less parallelism.


El sáb., 23 de feb. de 2019 a la(s) 21:27, Yeikel (

> I am following up on this question because I have a similar issue.
> When is that we need to control the parallelism manually? Skewed
> partitions?
> --
> Sent from:
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message