spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sung Hwan Chung <>
Subject Re: coalesce with shuffle or repartition is not necessarily fault-tolerant
Date Thu, 09 Oct 2014 06:51:50 GMT
Let's say you have some rows in a dataset (say X partitions initially).


You repartition to Y > X, then it seems that any of the following could be

partition 1             partition 2                ........................
A                          B
C                          E
D                           .
C                          E
A                          B
D                          .
D                          B
C                          E

etc. etc.

I.e., although each partition will have the same unordered set, the rows'
orders will change from call to call.

Now, because row ordering can change from call to call, if you do any
operation that depends on the order of items you saw, then lineage is no
longer deterministic. For example, it seems that the repartition call
itself is a row-order dependent call, because it creates a random number
generator with the partition index as the seed, and then call nextInt as
you go through the rows.

On Wed, Oct 8, 2014 at 10:14 PM, Patrick Wendell <> wrote:

> IIRC - the random is seeded with the index, so it will always produce
> the same result for the same index. Maybe I don't totally follow
> though. Could you give a small example of how this might change the
> RDD ordering in a way that you don't expect? In general repartition()
> will not preserve the ordering of an RDD.
> On Wed, Oct 8, 2014 at 3:42 PM, Sung Hwan Chung
> <> wrote:
> > I noticed that repartition will result in non-deterministic lineage
> because
> > it'll result in changed orders for rows.
> >
> > So for instance, if you do things like:
> >
> > val data = read(...)
> > val k = data.repartition(5)
> > val h = k.repartition(5)
> >
> > It seems that this results in different ordering of rows for 'k' each
> time
> > you call it.
> > And because of this different ordering, 'h' will result in different
> > partitions even, because 'repartition' distributes through a random
> number
> > generator with the 'index' as the key.

View raw message