spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Mocanu <amoc...@verticalscope.com>
Subject function state lost when next RDD is processed
Date Thu, 27 Mar 2014 14:04:14 GMT
Is there a way to pass a custom function to spark to run it on the entire stream? For example,
say I have a function which sums up values in each RDD and then across RDDs.

I've tried with map, transform, reduce. They all apply my sum function on 1 RDD. When the
next RDD comes the function starts from 0 so the sum of the previous RDD is lost.

Does Spark support a way of passing a custom function so that its state is preserved across
RDDs and not only within RDD?

Thanks
-Adrian


Mime
View raw message