flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Moiz Jinia <moiz.ji...@gmail.com>
Subject Re: Flink rolling upgrade support
Date Thu, 20 Jul 2017 04:14:14 GMT
No. This is the thread that answers my question -

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Does-FlinkKafkaConsumer010-care-about-consumer-group-td14323.html

Moiz

—
sent from phone

On 19-Jul-2017, at 10:04 PM, Ted Yu <yuzhihong@gmail.com> wrote:

This was the other thread, right ?

http://search-hadoop.com/m/Flink/VkLeQ0dXIf1SkHpY?subj=Re+Does+job+restart+resume+from+last+known+internal+checkpoint+

> On Wed, Jul 19, 2017 at 9:02 AM, Moiz Jinia <moiz.jinia@gmail.com> wrote:
> Yup! Thanks.
> 
> Moiz
> 
> —
> sent from phone
> 
> On 19-Jul-2017, at 9:21 PM, Aljoscha Krettek [via Apache Flink User Mailing List archive.]
<[hidden email]> wrote:
> 
> This was now answered in your other Thread, right?
> 
> Best,
> Aljoscha
> 
>>> On 18. Jul 2017, at 11:37, Moiz Jinia <[hidden email]> wrote:
>>> 
>>> Aljoscha Krettek wrote
>>> Hi,
>>> zero-downtime updates are currently not supported. What is supported in
>>> Flink right now is a savepoint-shutdown-restore cycle. With this, you
>>> first
>>> draw a savepoint (which is essentially a checkpoint with some meta data),
>>> then you cancel your job, then you do whatever you need to do (update
>>> machines, update Flink, update Job) and restore from the savepoint.
>>> 
>>> A possible solution for zero-downtime update would be to do a savepoint,
>>> then start a second Flink job from that savepoint, then shutdown the first
>>> job. With this, your data sinks would need to be able to handle being
>>> written to by 2 jobs at the same time, i.e. writes should probably be
>>> idempotent.
>>> 
>>> This is the link to the savepoint doc:
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/savepoints.html
>>> 
>>> Does that help?
>>> 
>>> Cheers,
>>> Aljoscha
>>> 
>>> On Fri, 16 Dec 2016 at 18:16 Andrew Hoblitzell &lt;
>> 
>>> ahoblitzell@
>> 
>>> &gt;
>>> wrote:
>>> 
>>>> Hi. Does Apache Flink currently have support for zero down time or the =
>>>> ability to do rolling upgrades?
>>>> 
>>>> If so, what are concerns to watch for and what best practices might =
>>>> exist? Are there version management and data inconsistency issues to =
>>>> watch for?=
>>>> 
>> 
>> When a second job instance is started in parallel from a savepoint, my
>> incoming kafka messages would get sharded between the 2 running instances of
>> the job (since they both would belong to the same consumer group). So when I
>> stop the older version of the job, i stand to lose data (inspite of the fact
>> that my downstream consumer is idempotent)
>> 
>> If I used a different consumer group for the new job version (and start it
>> from a savepoint), will the savepoint ensure that the second job instance
>> starts from the correct offset? Do I need to do anything extra to make this
>> work? (example set the uid on the source of the job).
>> 
>> Thanks!
>> Moiz
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-rolling-upgrade-support-tp10674p14313.html
>> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
> 
> 
> 
> If you reply to this email, your message will be added to the discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-rolling-upgrade-support-tp10674p14337.html
> To unsubscribe from Flink rolling upgrade support, click here.
> NAML
> 
> View this message in context: Re: Flink rolling upgrade support
> 
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.


Mime
View raw message