samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Venkatraman <jagadish1...@gmail.com>
Subject Re: checkpoint example?
Date Tue, 01 Mar 2016 22:57:07 GMT
Users need not worry about checkpointing. Samza will automatically commit
offsets every 60s. You can choose to commit more often by either
1. Setting task.commit.ms to a smaller value (or)
2. Doing manual commit yourself by setting task.commit.ms = -1. and calling
taskCoordinator.commit();

I'm curious as to Why processing from the exact previous offset is
unacceptable in your usecase?

Let's say you process till offfset 100, and crash. Should you not want to
resume from 100?







On Tue, Mar 1, 2016 at 1:41 PM, Jeff Ramin <jeff.ramin@singlewire.com>
wrote:

>
>
> On 03/01/2016 03:10 PM, Jagadish Venkatraman wrote:
>
>> You don't have to implement any state checkpoint. Samza automatically
>> checkpoints state for you. When you recover from a failure/restart you
>> will
>> resume processing from the previous checkpoint.
>>
> So, it's merely a configuration issue?
>
>   What's your usecase?
>>
>
> Pretty standard: have a consumer processing messages, which dies. When it
> comes back up,
> it needs to process messages not just from when it died, but perhaps 24
> hours prior to that time.
>
>
> --
> Jeff Ramin
> Software Engineer
> Singlewire Software
> 2601 W Beltline Hwy #510
> Madison, WI 53713
>
> Phone Direct - 608.661.1172
> www.singlewire.com
>
>


-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message