samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yan Fang <yanfang...@gmail.com>
Subject Re: Do we want to have the sliding window implementation?
Date Wed, 10 Sep 2014 18:32:07 GMT
@Mayur, thanks for the excellent explanation. :)

@Chris, yes, after second thought, I think we can achieve the sliding
window implementation with existing code base. Maybe adding a simple
example in hello-samza will be helpful since the implementation is not very
explicit but sliding window is a common use case.

Fang, Yan
yanfang724@gmail.com
+1 (206) 849-4108

On Wed, Sep 10, 2014 at 9:39 AM, Chris Riccomini <
criccomini@linkedin.com.invalid> wrote:

> Hey Guys,
>
> Our thought was that sliding window could be implemented with a buffer
> inside of a process() call. For example, you might have a list of 10
> elements, and every time process() is invoked, you could add the message
> to the ahead of the list, and dequeue the last element from the buffer (if
> it already has 10 elements in it).
>
> So, Samza currently doesn't support any explicit sliding window, but it
> seems to me that you could implement it in StreamTask.process() if you
> need to.
>
> Cheers,
> Chris
>
> On 9/9/14 10:31 PM, "Mayur Rustagi" <mayur.rustagi@gmail.com> wrote:
>
> >Slide window is another dimension to processing
> >
> >say my batch is 3 sec  & window is 9 sec then this is what I get
> >
> >[ x1 x2 x3][x4 x5 x6][x7 x8 x9]
> >
> >This is using slide but slide is equal to window size, but i may want to
> >get last 3 elements at any point of time then that would be
> >
> >
> >[ x1 x2 x3]  after3sec  [ x2 x3 x4] after3sec  [ x3 x4 x5] after3sec  [
> >x4 x5 x6]
> >
> >to implement this  you use batch of 3 sec, window of 9 sec & slide
> >duration of 3 sec. So we are sliding every 3 sec & also getting a batch
> >every 3 sec.
> >
> >
> >
> >
> >
> >--
> >Regards,
> >Mayur Rustagi
> >Ph: +1 (760) 203 3257
> >http://www.sigmoidanalytics.com
> >@mayur_rustagi
> >
> >On Wed, Sep 10, 2014 at 5:25 AM, Yan Fang <yanfang724@gmail.com> wrote:
> >
> >> Hi guys,
> >> realize that both Storm and Spark Streaming have sliding window
> >> implementation while Samza only has the fixed window (not sure if it's a
> >> correct name). I think you guys must consider this idea at the
> >>beginning of
> >> designing the Samza. What was the thought? Thank you.
> >> Cheers,
> >> Fang, Yan
> >> yanfang724@gmail.com
> >> +1 (206) 849-4108
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message