ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Shtykh <rsht...@yahoo.com.INVALID>
Subject Re: Improving Ignite window processing support
Date Sun, 27 Mar 2016 07:29:32 GMT
Dmitriy,
Honestly I was thinking only of window processing to cover only this specific area. But I
think looking at reactive streams makes sense and window processing can be implemented within
reactive streams with all three delivery guarantee semantics (starting with the easiest two).
Also I am interested in working on it. I will create a JIRA ticket for this and may have many
questions ;)
-Roman
 

    On Sunday, March 27, 2016 5:03 AM, Dmitriy Setrakyan <dsetrakyan@apache.org> wrote:
 

 Roman,

This is an excellent idea!

The API you are suggesting is very close to the reactive streams APIs [1],
so my suggestion would be to simply implement that API.

Another thing to focus on would be to provide guarantees:
- at least once
- at most once
- exactly once

I am not suggesting that we do it all in one step, but it would help to
identify what will be supported in the first iteration.

Do you mind taking a first step at this? We can discuss all questions
either here or in the chat room.

[1] http://www.reactive-streams.org/

D.

On Fri, Mar 25, 2016 at 8:08 PM, Murthy Kakarlamudi <ksatya@gmail.com>
wrote:

> Thanks for thinking about this feature. Due to lack of this feature, in our
> application we are currently planning to have some Timer based classes
> query the cache, perform aggregations and send the events to all the
> listeners.
> Having this feature built in into Ignite certainly helps.
>
> Satya.
> On Mar 25, 2016 10:38 PM, "Roman Shtykh" <rshtykh@yahoo.com.invalid>
> wrote:
>
> > Igniters,
> > I was thinking about improving Ignite window processing support so that
> > queries for windowed data is not user-initiated (which can have timing
> > issues) but rather event-driven, similar to ContinuousQuery but being
> able
> > to fire size-based or time-based entry windows to the listening client.
> > Some thougths on implementation:1. Guarantee all window events go to the
> > same partition (user will need to specify it as a windowed cache). Maybe
> it
> > can be rather implemented by extending IgniteQueue?2. Set a trigger on
> > cache, which will listen to eviction (in case of size-based windowed
> cache)
> > or expiration (time-based cache) events of the cache and fire entries.
> > It has to be exposed to the user by some API, very roughly something
> > likeIgniteStream<K,V> is = IgniteStream.on(cache).with(windowingType,
> > filterPredidate).aggregate(aggFunc)is.run() // to continuously return
> > windowed results
> > where windowingType is time/size/session-based windows, aggFunc is
> > sum/min/max/etc. or user-specified.
> > It can be an experimental feature and I think it is a useful API to
> > enforce our stream processing, but I would like to know your opinion. Do
> > you think such API is needed? I have a limited knowledge of Ignite
> > internals -- any other ideas on the implementation?
> > For your reference,
> > https://flink.apache.org/news/2015/12/04/Introducing-windows.html is a
> > good introduction on windows processing.
> > -Roman
> >
> > P.S. Other things like operator/event/ingress time but has to be
> > considered for the implementation are omitted.
> >
>


   
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message