spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sung Hwan Chung <coded...@cs.stanford.edu>
Subject Is RDD partition index consistent?
Date Mon, 06 Oct 2014 19:33:22 GMT
Is the RDD partition index you get when you call mapPartitionWithIndex
consistent under fault-tolerance condition?

I.e.

1. Say index is 1 for one of the partitions when you call
data.mapPartitionWithIndex((index, rows) => ....) // Say index is 1
2. The partition fails (maybe a long with a bunch of other partitions).
3. When the partitions get restarted somewhere else, will they retain the
same index value, as well as all the lineage arguments?

Mime
View raw message