samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Di Spaltro <dan.dispal...@gmail.com>
Subject Re: Do we want to have the sliding window implementation?
Date Wed, 10 Sep 2014 18:54:24 GMT
That's very similar to how Esper implements a sliding window.  They have
the concept of oldEvents and newEvents that get passed in to every
"process". oldEvents would have the old events leaving a window and
newEvents would have the new ones coming in.  The difference here is that
esper provides the sliding window capability, but it can also be helpful to
translate domain problems to your code[1].

[1]
http://esper.codehaus.org/esper-5.0.0/doc/reference/en-US/html_single/index.html#view-win-time


On Wed, Sep 10, 2014 at 9:39 AM, Chris Riccomini <
criccomini@linkedin.com.invalid> wrote:

> Hey Guys,
>
> Our thought was that sliding window could be implemented with a buffer
> inside of a process() call. For example, you might have a list of 10
> elements, and every time process() is invoked, you could add the message
> to the ahead of the list, and dequeue the last element from the buffer (if
> it already has 10 elements in it).
>
> So, Samza currently doesn't support any explicit sliding window, but it
> seems to me that you could implement it in StreamTask.process() if you
> need to.
>
> Cheers,
> Chris
>
> On 9/9/14 10:31 PM, "Mayur Rustagi" <mayur.rustagi@gmail.com> wrote:
>
> >Slide window is another dimension to processing
> >
> >say my batch is 3 sec  & window is 9 sec then this is what I get
> >
> >[ x1 x2 x3][x4 x5 x6][x7 x8 x9]
> >
> >This is using slide but slide is equal to window size, but i may want to
> >get last 3 elements at any point of time then that would be
> >
> >
> >[ x1 x2 x3]  after3sec  [ x2 x3 x4] after3sec  [ x3 x4 x5] after3sec  [
> >x4 x5 x6]
> >
> >to implement this  you use batch of 3 sec, window of 9 sec & slide
> >duration of 3 sec. So we are sliding every 3 sec & also getting a batch
> >every 3 sec.
> >
> >
> >
> >
> >
> >--
> >Regards,
> >Mayur Rustagi
> >Ph: +1 (760) 203 3257
> >http://www.sigmoidanalytics.com
> >@mayur_rustagi
> >
> >On Wed, Sep 10, 2014 at 5:25 AM, Yan Fang <yanfang724@gmail.com> wrote:
> >
> >> Hi guys,
> >> realize that both Storm and Spark Streaming have sliding window
> >> implementation while Samza only has the fixed window (not sure if it's a
> >> correct name). I think you guys must consider this idea at the
> >>beginning of
> >> designing the Samza. What was the thought? Thank you.
> >> Cheers,
> >> Fang, Yan
> >> yanfang724@gmail.com
> >> +1 (206) 849-4108
>
>


-- 
Dan Di Spaltro

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message