spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Re: Persist and unpersist
Date Tue, 27 May 2014 17:43:30 GMT
Daniel,

Is SPARK-1103 <https://issues.apache.org/jira/browse/SPARK-1103> related to
your example? Automatic unpersist()-ing of unreferenced RDDs would be nice.

Nick
‚Äč


On Tue, May 27, 2014 at 12:28 PM, Daniel Darabos <
daniel.darabos@lynxanalytics.com> wrote:

> I keep bumping into a problem with persisting RDDs. Consider this (silly)
> example:
>
> def everySecondFromBehind(input: RDD[Int]): RDD[Int] = {
>   val count = input.count
>   if (count % 2 == 0) {
>     return input.filter(_ % 2 == 1)
>   } else {
>     return input.filter(_ % 2 == 0)
>   }
> }
>
>
> The situation is that we want to do two things with an RDD (a "count" and
> a "filter" in the example). The "input" RDD may represent a very expensive
> calculation. So it would make sense to add an "input.cache()" line at the
> beginning. But where do we put "input.unpersist()"?
>
> input.cache()val count = input.countval result = input.filter(...)
> input.unpersist()return result
>
>
> "input.filter()" is lazy, so this does not work as expected. We only want
> to release "input" from the cache once nothing depends on it anymore. Maybe
> "result" was garbage collected. Maybe "result" itself has been cached. But
> there is no way to detect such conditions.
>
> Our current approach is to just leave the RDD cached, and it will get
> dumped at some point anyway. Is there a better solution? Thanks for any
> tips.
>

Mime
View raw message