spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <>
Subject Need some Clarification on checkpointing w.r.t Spark Structured Streaming
Date Mon, 11 Sep 2017 09:36:50 GMT
Hi All,

I was wondering if we need to checkpoint both read and write streams when
reading from Kafka and inserting into a target store?

for example

sparkSession.readStream().option("checkpointLocation", "hdfsPath").load()


dataSet.writeStream().option("checkpointLocation", "hdfsPath")


View raw message