spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yan Fang <yanfang...@gmail.com>
Subject Does RDD checkpointing store the entire state in HDFS?
Date Thu, 17 Jul 2014 00:38:06 GMT
Hi guys,

am wondering how the RDD checkpointing
<https://spark.apache.org/docs/latest/streaming-programming-guide.html#RDD
Checkpointing> works in Spark Streaming. When I use updateStateByKey, does
the Spark store the entire state (at one time point) into the HDFS or only
put the transformation into the HDFS? Thank you.

Best,

Fang, Yan
yanfang724@gmail.com
+1 (206) 849-4108

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message