spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earthson <>
Subject Re: Incredible slow iterative computation
Date Tue, 06 May 2014 01:35:01 GMT
checkpoint seems to be just add a CheckPoint mark? You need an action after
marked it. I have tried it with success:)

newRdd =
newRdd.checkpoint // <<checkpoint here
newRdd.isCheckpointed // false here
newRdd.foreach(x => {}) // Force evaluation
newRdd.isCheckpointed // true here


If you have new broadcast object for each step of iteration, broadcast will
eat up all of the memory. You may need to set "spark.cleaner.ttl" to a small
enough value.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message