ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Apache Flink <=> Apache Ignite integration
Date Mon, 04 Apr 2016 14:54:01 GMT
Hi!

  - Sounds like the having Ignite for snapshots should work pretty much out
of the box (via the IGFS)
  - The source and sink connector sounds like the next logical step. Does
Ignite have a notion of stream partitions and offsets, to build a
consistent replay around? This should probably have its dedicated issue and
discussion thread.

  - For Ignite as an execution backend - I am not sure how relevant and
feasible that is. Many DataStream API features make use of the specific
Flink runtime. For streaming, the runtime is not as decoupled as for batch.
  - I think the parameter server integration would not be part of the Flink
codebase - this is a pretty application specific thing that should be its
own project and it is actually not tightly coupled to Flink.

Greetings,
Stephan


On Mon, Apr 4, 2016 at 4:35 PM, Robert Metzger <rmetzger@apache.org> wrote:

> Hi Raul,
>
> thanks a lot for reaching out to the Flink community.
> I'm really excited to see a Flink connector in Ignite. If you feel that the
> connector would be more suitable for our "connector library" feel free to
> open a JIRA and open a pull request.
>
> Were there requests in the Ignite community to have an integration with
> Flink?
>
>
>
> On Thu, Mar 31, 2016 at 5:20 PM, Saikat Maitra <saikat.maitra@gmail.com>
> wrote:
>
> > Hi ,
> >
> > I agree with Roman and Raul.
> > https://issues.apache.org/jira/browse/IGNITE-813 allows injecting data
> to
> > into cache via Data Streamer. Integrating with Ignite FileSystem for
> source
> > and sink will allow for bidirectional connector. It will also allow
> easier
> > implementation for DataStream transformations over Ignite FileSystem.
> >
> > Regards
> > Saikat
> >
> > On Thu, Mar 31, 2016 at 2:44 PM, Aljoscha Krettek <aljoscha@apache.org>
> > wrote:
> >
> > > Hi,
> > > it should already be possible to use the Ignite FileSystem to store
> state
> > > since we just use the HDFS FileSystem interface for that. Of course,
> one
> > > would have to properly set up the jars and paths and everything for
> Flink
> > > to pick up the IGFS classes.
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > On Wed, 30 Mar 2016 at 16:50 Raul Kripalani <raulk@apache.org> wrote:
> > >
> > > > On Wed, Mar 30, 2016 at 2:20 PM, Roman <rshtykh@yahoo.com.invalid>
> > > wrote:
> > > >
> > > > > Raul,
> > > > >
> > > > > Small comment from me.
> > > > >
> > > > > >* As a Flink sink => inject data directly into a cache via
a
> > > > DataStreamer.
> > > > > After reviews, IGNITE-813 is exactly this functionality.
> > > > >
> > > > >
> > > > That's cool, Roman! The idea would be to host these (richer) modules
> as
> > > > Flink connectors, like they do with others:
> > > >
> > > >
> https://github.com/apache/flink/tree/master/flink-streaming-connectors
> > > > https://github.com/apache/flink/tree/master/flink-batch-connectors
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message