spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: spark challenge: zip with next???
Date Fri, 30 Jan 2015 00:36:18 GMT
Hi,

On Fri, Jan 30, 2015 at 6:32 AM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com>
wrote:

>  Make a copy of your RDD with an extra entry in the beginning to offset.
> The you can zip the two RDDs and run a map to generate an RDD of
> differences.
>

Does that work? I recently tried something to compute differences between
each entry and the next, so I did
  val rdd1 = ... // null element + rdd
  val rdd2 = ... // rdd + null element
but got an error message about zip requiring data sizes in each partition
to match.

Tobias

Mime
View raw message