samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Ramin <jeff.ra...@singlewire.com>
Subject Re: checkpoint example?
Date Wed, 02 Mar 2016 21:47:45 GMT

Let me explain the use case in more detail:

We are keeping the data in 2 systems in sync. Let's name the upstream 
system as the "source"
and the downstream system as "destination". The destination system is 
backed up (locally)
once per day (let's say at 3:00 AM).

It's now 1:00 PM, and we've been processing messages to keep the systems 
in sync since 3:00 AM.
At this time, the destination system dies. A replacement is created, and 
the backup taken at 3:00 AM
is used as the starting state for the new system.

We'd then want to replay all messages that occurred since 3:00.


On 03/01/2016 04:57 PM, Jagadish Venkatraman wrote:
> Users need not worry about checkpointing. Samza will automatically commit
> offsets every 60s. You can choose to commit more often by either
> 1. Setting task.commit.ms to a smaller value (or)
> 2. Doing manual commit yourself by setting task.commit.ms = -1. and calling
> taskCoordinator.commit();
>
> I'm curious as to Why processing from the exact previous offset is
> unacceptable in your usecase?
>
> Let's say you process till offfset 100, and crash. Should you not want to
> resume from 100?
>
>
>
>
>
>
>
> On Tue, Mar 1, 2016 at 1:41 PM, Jeff Ramin <jeff.ramin@singlewire.com>
> wrote:
>
>>
>> On 03/01/2016 03:10 PM, Jagadish Venkatraman wrote:
>>
>>> You don't have to implement any state checkpoint. Samza automatically
>>> checkpoints state for you. When you recover from a failure/restart you
>>> will
>>> resume processing from the previous checkpoint.
>>>
>> So, it's merely a configuration issue?
>>
>>    What's your usecase?
>> Pretty standard: have a consumer processing messages, which dies. When it
>> comes back up,
>> it needs to process messages not just from when it died, but perhaps 24
>> hours prior to that time.
>>
>>
>> --
>> Jeff Ramin
>> Software Engineer
>> Singlewire Software
>> 2601 W Beltline Hwy #510
>> Madison, WI 53713
>>
>> Phone Direct - 608.661.1172
>> www.singlewire.com
>>
>>
>

-- 
Jeff Ramin
Software Engineer
Singlewire Software
2601 W Beltline Hwy #510
Madison, WI 53713

Phone Direct - 608.661.1172
www.singlewire.com


Mime
View raw message