spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Kronenfeld <nkronenf...@oculusinfo.com>
Subject Re: Visitor function to RDD elements
Date Tue, 22 Oct 2013 22:25:54 GMT
You shouldn't have to fly data around

You can just run it first on partition 0, then on partition 1, etc...  I
may have the name slightly off, but something approximately like:

for (p <- 0 until numPartitions)
  data.mapPartitionsWithIndex((i, iter) => if (0 == p) iter.map(fcn) else
List().iterator)

should work... BUT that being said, you've now really lost the point of
using Spark to begin with.

Mime
View raw message