spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Restart streaming query spark 2.1 structured streaming
Date Tue, 15 Aug 2017 19:25:57 GMT
See
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing

Though I think that this currently doesn't work with the console sink.

On Tue, Aug 15, 2017 at 9:40 AM, purna pradeep <purna2pradeep@gmail.com>
wrote:

> Hi,
>
>>
>> I'm trying to restart a streaming query to refresh cached data frame
>>
>> Where and how should I restart streaming query
>>
>
>
> val sparkSes = SparkSession
>
>       .builder
>
>       .config("spark.master", "local")
>
>       .appName("StreamingCahcePoc")
>
>       .getOrCreate()
>
>
>
>     import sparkSes.implicits._
>
>
>
>     val dataDF = sparkSes.readStream
>
>       .schema(streamSchema)
>
>       .csv("testData")
>
>
>
>
>
>        val query = counts.writeStream
>
>       .outputMode("complete")
>
>       .format("console")
>
>       .start()
>
>
> query.awaittermination()
>
>
>
>>
>>
>>

Mime
View raw message