spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Monchy <nico...@gumgum.com>
Subject Re: Spark Streaming checkpoints and code upgrade
Date Wed, 09 Sep 2015 20:42:36 GMT
Ok. Thank you for the reply.

On Wed, Sep 9, 2015 at 11:40 AM, Tathagata Das <tdas@databricks.com> wrote:

> Its pretty much impossible to do across arbitrary code changes. For that,
> the best way is to go forward is the store and load the offsets yourselves.
>
> On Wed, Sep 9, 2015 at 10:19 AM, Nicolas Monchy <nicolas@gumgum.com>
> wrote:
>
>> Hello,
>>
>> I am using Spark Streaming and the Kafka Direct API and I am
>> checkpointing the metadata.
>> Checkpoints aren't recoverable if you upgrade code so I am losing the
>> last consumed offsets in this case.
>>
>> I know I can build a system to store and load the offsets for each batch
>> but before implementing that I would like to know if checkpoints are going
>> to be able to recover a code upgrade in a foreseeable future ?
>>
>> Thanks,
>> Nicolas
>>
>
>


-- 
Nicolas Monchy  |  Software Engineer, Big Data
*GumGum* <http://www.gumgum.com>  |  *Ads that stick*
424 375 8823  |  nicolas@gumgum.com

Mime
View raw message