spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Fregly <ch...@fregly.com>
Subject Re: Checkpoint Vs Cache
Date Sat, 03 May 2014 06:19:49 GMT
http://docs.sigmoidanalytics.com/index.php/Checkpoint_and_not_running_out_of_disk_space


On Mon, Apr 14, 2014 at 2:43 AM, Cheng Lian <lian.cs.zju@gmail.com> wrote:

> Checkpointed RDDs are materialized on disk, while cached RDDs are
> materialized in memory. When memory is insufficient, cached RDD blocks (1
> block per partition) will be evicted in an LRU manner. An evicted RDD block
> will be spilled to disk if the storage level of the RDD allows, otherwise
> this block vanishes entirely and must be recomputed from the lineage DAG if
> it's referenced later.
>
>
> On Mon, Apr 14, 2014 at 10:20 AM, David Thomas <dt5434884@gmail.com>wrote:
>
>> What is the difference between checkpointing and caching an RDD?
>>
>
>

Mime
View raw message