spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <t...@databricks.com>
Subject Re: Spark Streaming checkpoints and code upgrade
Date Wed, 09 Sep 2015 18:40:14 GMT
Its pretty much impossible to do across arbitrary code changes. For that,
the best way is to go forward is the store and load the offsets yourselves.

On Wed, Sep 9, 2015 at 10:19 AM, Nicolas Monchy <nicolas@gumgum.com> wrote:

> Hello,
>
> I am using Spark Streaming and the Kafka Direct API and I am checkpointing
> the metadata.
> Checkpoints aren't recoverable if you upgrade code so I am losing the last
> consumed offsets in this case.
>
> I know I can build a system to store and load the offsets for each batch
> but before implementing that I would like to know if checkpoints are going
> to be able to recover a code upgrade in a foreseeable future ?
>
> Thanks,
> Nicolas
>

Mime
View raw message