spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antonin Delpeuch <>
Subject Re: RDD order guarantees
Date Tue, 07 Apr 2020 14:25:53 GMT

Sorry to dig out this thread but this bug is still present.

The fix proposed in this thread (creating a new FileSystem implementation
which sorts listed files) was rejected, with the suggestion that it is the
FileInputFormat's responsibility to sort the file names if preserving
partition order is desired:

Given that Spark RDDs are supposed to preserve the order of the collections
they represent, this would still deserve to be fixed in Spark, I think. As a
user, I expect that if I use saveAsTextFile and then load the resulting file
with sparkContext.textFile, I obtain a dataset in the same order.

Because Spark uses the FileInputFormats exposed by Hadoop, that would mean
either patching Hadoop for it to sort file names directly (which is likely
going to fail since Hadoop might not care about the ordering in general), or
create subclasses of all Hadoop formats used in Spark, adding the required
sorting to the listStatus method. This strikes me as less elegant than
implementing a new FileSystem as suggested by Reynold, though.

Another way to "fix" this would be to mention in the docs that order is not
preserved in this scenario, which could hopefully avoid bad surprises to
others (just like we already have a caveat about nondeterminism of order
after shuffles).

I would be happy to try submitting a fix for this, if there is a consensus
around the correct course of action.


Sent from:

To unsubscribe e-mail:

View raw message