spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iulian DragoČ™ <iulian.dra...@typesafe.com>
Subject Re: Is coalesce smart while merging partitions?
Date Thu, 08 Oct 2015 10:17:06 GMT
It's smart. Have a look at
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala#L123

On Thu, Oct 8, 2015 at 4:00 AM, Cesar Flores <cesar7@gmail.com> wrote:

> It is my understanding that the default behavior of coalesce function when
> the user reduce the number of partitions is to only merge them without
> executing shuffle.
>
> My question is: Is this merging smart? For example does spark try to merge
> the small partitions first or the election of partitions to merge is random?
>
>
> Thanks
> --
> Cesar Flores
>



-- 

--
Iulian Dragos

------
Reactive Apps on the JVM
www.typesafe.com

Mime
View raw message