spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From M Singh <mans2si...@yahoo.com.INVALID>
Subject Apache Spark Structured Streaming - Kafka Streaming - Option to ignore checkpoint
Date Fri, 23 Mar 2018 04:11:33 GMT
Hi:
I am working on a realtime application using spark structured streaming (v 2.2.1). The application
reads data from kafka and if there is a failure, I would like to ignore the checkpoint. 
Is there any configuration to just read from last kafka offset after a failure and ignore
any offset checkpoints ? 
Also, I believe that the checkpoint also saves state and will continue to aggregations after
recovery.  Is there any way to ignore checkpointed state ?
Also, is there a way to selectively save state or offset checkpoint only ?

Thanks

Mime
View raw message