spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vincent gromakowski <vincent.gromakow...@gmail.com>
Subject Re: How to update structured streaming apps gracefully
Date Tue, 18 Dec 2018 17:45:19 GMT
Checkpointing is only used for failure recovery not for app upgrades. You
need to manually code the unload/load and save it to a persistent store

Le mar. 18 déc. 2018 à 17:29, Priya Matpadi <pmatpadi@gmail.com> a écrit :

> Using checkpointing for graceful updates is my understanding as well,
> based on the writeup in
> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing,
> and some prototyping. Have you faced any missed events?
>
> On Mon, Dec 17, 2018 at 6:56 PM Yuta Morisawa <
> yu-morisawa@kddi-research.jp> wrote:
>
>> Hi
>>
>> Now I'm trying to update my structured streaming application.
>> But I have no idea how to update it gracefully.
>>
>> Should I stop it, replace a jar file then restart it?
>> In my understanding, in that case, all the state will be recovered if I
>> use checkpoints.
>> Is this correct?
>>
>> Thank you,
>>
>>
>> --
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>

Mime
View raw message