spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From brettplarson <brettpatricklar...@gmail.com>
Subject Impact of .localCheckpoint() and executor dying
Date Wed, 06 Jan 2021 17:14:54 GMT
Hello,
I am wondering what the impact of using .localCheckpoint() and having the
executor die would be? 

My understanding is that .localCheckpoint() breaks the lineage of the RDD
and this requires that the entire RDD to be rebuild instead of being able to
recompute lost partitions.

Does each executor store a copy of the entire RDD?

It's unclear to me the benefit of using Checkpoint over .localCheckpoint. (I
am aware that this is HDFS backed, but it's unclear the implications of
this)

Please let me know,
Thank you!




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message