spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earthson <>
Subject Cache issue for iteration with broadcast
Date Mon, 05 May 2014 03:15:15 GMT
A new broadcast object will generated for every iteration step, it may eat up
the memory and make persist fail. 

The broadcast object should not be removed because RDD may be recomputed.
And I am trying to prevent recomputing RDD, it need old broadcast release
some memory.

I've tried to set "spark.cleaner.ttl", but my task runs into Error(broadcast
object not found), I think task is recomputed. I don't think this is a good
idea, it makes my code depends on my environment much more.

So I changed the persistLevel of my RDD to MEMORY_AND_DISK, but it runs into
ERROR(broadcast object not found) too.  And I remove the setting of
spark.cleaner.ttl, finally.

I think support for cache should be more friendly, and broadcast object
should be cached, too. flush the old object to disk is much more essential
than the new one.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message