spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Monchy <>
Subject Spark Streaming checkpoints and code upgrade
Date Wed, 09 Sep 2015 17:19:05 GMT

I am using Spark Streaming and the Kafka Direct API and I am checkpointing
the metadata.
Checkpoints aren't recoverable if you upgrade code so I am losing the last
consumed offsets in this case.

I know I can build a system to store and load the offsets for each batch
but before implementing that I would like to know if checkpoints are going
to be able to recover a code upgrade in a foreseeable future ?


View raw message