spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earthson <>
Subject Re: Cache issue for iteration with broadcast
Date Mon, 05 May 2014 17:09:35 GMT
Yes, I've tried.

The problem is new broadcast object generated by every step until eat up all
of the memory. 

I solved it by using RDD.checkpoint to remove dependences to old broadcast
object, and use cleanner.ttl to clean up these broadcast object

If there's more elegant way to solve this problem, please tell me:) 

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message