spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Rosen <rosenvi...@gmail.com>
Subject Re: JavaPairRDD unpersist
Date Wed, 23 Oct 2013 21:19:28 GMT
This is a bug; all three Java*RDD classes should have had unpersist()
methods added to them, instead of just JavaRDD (
https://github.com/apache/incubator-spark/commit/4a318774088f829fe54c3ef0b5f565a845631b4e).
 I'll submit a pull request to fix this.  In 0.8, you can access the
underlying Scala RDD and call unpersist() on that:

javaPairRDD.rdd().unpersist()

If you want to know the details of why we need to add unpersist() to each
Java*RDD class, rather than adding it once in JavaRDDLike, check out the
"implementation traits" section of
https://cwiki.apache.org/confluence/display/SPARK/Java+API+Internals

- Josh



On Wed, Oct 23, 2013 at 2:01 PM, Yann Luppo <YannLuppo@livenation.com>wrote:

>  Hi,
>
>  I've noticed that in 0.8.0 JavaRDD got a new "unpersist" method. I was
> wondering if this method was going to be ported eventually to JavaPairRDD.
> Is this somewhere on the road map? Or should we just change all of our
> JavaPairRDD to JavaRDDs w/ tuples for now?
>
>  Thanks,
> Yann
>

Mime
View raw message