samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik Sriram <amaron...@gmail.com>
Subject Re: Windowing Guarantees in samza
Date Sun, 15 Feb 2015 20:48:57 GMT
Hi Benjamin,
  I'm trying to employ samza for a similar use case and this is what I did
to mitigate this:
1> I have a notion of timestamp in the messages itself that I listen to.
This way, as I get messages, I can maintain state by time period of
aggregation by attaching the time period to the key of my state store. Ex.,
if I want hourly aggregations,
I store keys by getHourFromTimestamp(message.timestamp):message.key ->
state .
2> I configure a window every 10/20 mins and using samza's KVStore's
range() iterator, combined with Kafka's guaranteed in order delivery of
messages, I can ensure to flush out only aggregates of previous timestamps
once I have seen atleast one message from the next timeperiod.

I saw various issues with having control messages to dictate end of time
periods like when you have multiple producers producing to your given input
topic, how would you ensure your control message appears after all the
producers have produced for that time period? Or if the control message
happens to be lost etc., how would you ensure reliability? I have found
control messages more useful only for adhoc aggregations that may depend on
factors other than time or size of batch. but of course, it all depends on
use case.

I'm also interested to know if anyone else has solved this problem with any
other method?

Thanks,
Karthik


Thanks,
Karthik

On Sun, Feb 15, 2015 at 10:51 AM, Benjamin Edwards <edwards.benj@gmail.com>
wrote:

> Hi
>
> Based on what I can see in the run loop class, there are a few things that
> seem a little problematic for windowed processing with respect to time:
>
> 1) No ability to schedule *when* on an interval you might start. For
> instance, if you wanted to process a window on the hour, every hour, there
> is no way to do this.
>
> 2) You don't get passed the time. I guess this is simply due to the fact
> that the window isn't really trying to keep up, or pin itself to a given
> phase. If you get behind, well tough. You just added some phase to your
> series.
>
> What do people normally do to mitigate this? I was thinking that rather
> than using the Windowed task I would simply have the producer use a timer
> and once a period send a control message with the time stamp. This would
> indicate to my task that period was up and state should be flushed to db,
> aggregated to another stream etc..
>
> Note that I am not trying to do real time processing with hard constraints,
> or anything like that, I just need things that mostly happened within a
> given frame to get grouped and most importantly for things to happen "on
> the minute" or "on the hour" etc.
>
> Ben
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message