spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yann Luppo <YannLu...@LiveNation.com>
Subject Re: JavaPairRDD unpersist
Date Wed, 23 Oct 2013 22:10:53 GMT
Perfect! Thank you Josh.

From: Josh Rosen <rosenville@gmail.com<mailto:rosenville@gmail.com>>
Reply-To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>"
<user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>>
Date: Wednesday, October 23, 2013 2:19 PM
To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>" <user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>>
Subject: Re: JavaPairRDD unpersist

This is a bug; all three Java*RDD classes should have had unpersist() methods added to them,
instead of just JavaRDD (https://github.com/apache/incubator-spark/commit/4a318774088f829fe54c3ef0b5f565a845631b4e).
 I'll submit a pull request to fix this.  In 0.8, you can access the underlying Scala RDD
and call unpersist() on that:

javaPairRDD.rdd().unpersist()

If you want to know the details of why we need to add unpersist() to each Java*RDD class,
rather than adding it once in JavaRDDLike, check out the "implementation traits" section of
https://cwiki.apache.org/confluence/display/SPARK/Java+API+Internals

- Josh



On Wed, Oct 23, 2013 at 2:01 PM, Yann Luppo <YannLuppo@livenation.com<mailto:YannLuppo@livenation.com>>
wrote:
Hi,

I've noticed that in 0.8.0 JavaRDD got a new "unpersist" method. I was wondering if this method
was going to be ported eventually to JavaPairRDD.
Is this somewhere on the road map? Or should we just change all of our JavaPairRDD to JavaRDDs
w/ tuples for now?

Thanks,
Yann


Mime
View raw message