spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liang-Chi Hsieh <>
Subject Re: How to cache SparkPlan.execute for reusing?
Date Fri, 03 Mar 2017 08:37:17 GMT

Not sure what you mean in "its parents have to reuse it by creating new

As SparkPlan.execute returns new RDD every time, you won't expect the cached
RDD can be reused automatically, even you reuse the SparkPlan in several

Btw, is there any existing ways to reuse SparkPlan?

summerDG wrote
> Thank you very much. The reason why the output is empty is that the query
> involves join. I forgot to mention it in the question. So even I succeed
> in caching the RDD, the following SparkPlans in the query will not reuse
> it.
> If there is a SparkPlan of the query, which has several "parent" nodes,
> its "parents" have to reuse it by creating new RDDs?

Liang-Chi Hsieh | @viirya 
Spark Technology Center 
View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe e-mail:

View raw message