spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject [SS] Console sink not supporting recovering from checkpoint location? Why?
Date Mon, 07 Aug 2017 09:58:37 GMT

While exploring checkpointing with kafka source and console sink I've
got the exception:

// today's build from the master
scala> spark.version
res8: String = 2.3.0-SNAPSHOT

scala> val q = records.
     |   writeStream.
     |   format("console").
     |   option("truncate", false).
     |   option("checkpointLocation", "/tmp/checkpoint"). // <--
checkpoint directory
     |   trigger(Trigger.ProcessingTime(10.seconds)).
     |   outputMode(OutputMode.Update).
     |   start
org.apache.spark.sql.AnalysisException: This query does not support
recovering from checkpoint location. Delete /tmp/checkpoint/offsets to
start over.;
  at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:222)
  at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:278)
  at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:284)
  ... 61 elided

The "trigger" is the change and this line in

Why is this needed? I can't think of a use case where console sink
could not recover from checkpoint location (since all the information
is available). I'm lost on it and would appreciate some help (to
recover :))

Jacek Laskowski
Mastering Apache Spark 2
Follow me at

To unsubscribe e-mail:

View raw message