samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com.INVALID>
Subject Re: Do we want to have the sliding window implementation?
Date Wed, 10 Sep 2014 16:39:55 GMT
Hey Guys,

Our thought was that sliding window could be implemented with a buffer
inside of a process() call. For example, you might have a list of 10
elements, and every time process() is invoked, you could add the message
to the ahead of the list, and dequeue the last element from the buffer (if
it already has 10 elements in it).

So, Samza currently doesn't support any explicit sliding window, but it
seems to me that you could implement it in StreamTask.process() if you
need to.

Cheers,
Chris

On 9/9/14 10:31 PM, "Mayur Rustagi" <mayur.rustagi@gmail.com> wrote:

>Slide window is another dimension to processing
>
>say my batch is 3 sec  & window is 9 sec then this is what I get
>
>[ x1 x2 x3][x4 x5 x6][x7 x8 x9]
>
>This is using slide but slide is equal to window size, but i may want to
>get last 3 elements at any point of time then that would be
>
>
>[ x1 x2 x3]  after3sec  [ x2 x3 x4] after3sec  [ x3 x4 x5] after3sec  [
>x4 x5 x6]
>
>to implement this  you use batch of 3 sec, window of 9 sec & slide
>duration of 3 sec. So we are sliding every 3 sec & also getting a batch
>every 3 sec. 
>
>
>
>
>
>-- 
>Regards,
>Mayur Rustagi
>Ph: +1 (760) 203 3257
>http://www.sigmoidanalytics.com
>@mayur_rustagi
>
>On Wed, Sep 10, 2014 at 5:25 AM, Yan Fang <yanfang724@gmail.com> wrote:
>
>> Hi guys,
>> realize that both Storm and Spark Streaming have sliding window
>> implementation while Samza only has the fixed window (not sure if it's a
>> correct name). I think you guys must consider this idea at the
>>beginning of
>> designing the Samza. What was the thought? Thank you.
>> Cheers,
>> Fang, Yan
>> yanfang724@gmail.com
>> +1 (206) 849-4108


Mime
View raw message