ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: [DISCUSS] Pub/Sub Streamer Implementation
Date Fri, 22 Nov 2019 20:44:48 GMT
Hi Emmanouil,

Do you have any questions or need any support from the community?

-
Denis


On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra <saikat.maitra@gmail.com>
wrote:

> Hi Emmanouil,
>
> Can you please take a looks at dev utils, if this is something you are
> looking for https://github.com/apache/ignite/tree/master/modules/dev-utils
> ?
>
> IMO, if you can release Pub/Sub server in maven and then use it as
> dependency, that would be great.
>
> Regards,
> Saikat
>
> On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra <saikat.maitra@gmail.com>
> wrote:
>
> > Hi Emmanouil,
> >
> > The master branch has been initialized. I have raised a PR to migrate
> > Flink module to ignite-extensions.
> >
> > PR https://github.com/apache/ignite-extensions/pull/1
> > Jira : https://issues.apache.org/jira/browse/IGNITE-12356
> >
> > Regards,
> > Saikat
> >
> >
> > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras <
> > gkatzioura@gmail.com> wrote:
> >
> >> Hi all!
> >>
> >> This is an update. The integration is tested with mocks based on the
> >> original Ignite repo [1].
> >> I am currently waiting for the first commits on the new integration
> >> project
> >> to take place in order to create my pull request accordingly to the
> >> initial
> >> commits.
> >> In order to simulate the Pub/Sub server in a test I used a test server
> >> implementation [2]. Unfortunately they don't have this util uploaded in
> a
> >> maven repo, thus it needs to be hosted on a maven repo (Apache License
> >> 2.0). Is there a repo for utilities that Ignite uses for the test/build?
> >> I have also created a ticket [3]
> >>
> >> Kind regards
> >>
> >> [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> >> [2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator
> >> [3] https://issues.apache.org/jira/browse/IGNITE-12262
> >>
> >> *Emmanouil Gkatziouras*
> >> https://egkatzioura.com/ |
> >> https://www.linkedin.com/in/gkatziourasemmanouil/
> >> https://github.com/gkatzioura
> >>
> >>
> >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra <saikat.maitra@gmail.com>
> >> wrote:
> >>
> >> > Hello Ilya,
> >> >
> >> > Thank you for your email. Yes, before we remove any integration we
> will
> >> > move them to another repository and release.
> >> >
> >> > We are thinking we will start with Pub/Sub Streamer implementation and
> >> > continue to move other integration as mentioned in the following doc.
> >> >
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >> >
> >> > Regards,
> >> > Saikat
> >> >
> >> >
> >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev <
> >> > ilya.kasnacheev@gmail.com>
> >> > wrote:
> >> >
> >> > > Hello!
> >> > >
> >> > > My take on this: we want to offload some of integrations, but please
> >> > commit
> >> > > and release them to another project first, and only then we will
> >> decide
> >> > to
> >> > > remove them from our repository.
> >> > >
> >> > > Please also make sure they are available with same artifact name and
> >> API.
> >> > > If this has to be changed, let's wait for AI 3.0
> >> > >
> >> > > Regards,
> >> > > --
> >> > > Ilya Kasnacheev
> >> > >
> >> > >
> >> > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <saikat.maitra@gmail.com
> >:
> >> > >
> >> > > > Hello Emmanouil,
> >> > > >
> >> > > > As discussed earlier, I discussed with Apache Bahir community
and
> >> they
> >> > > are
> >> > > > interested to have Apache Ignite extensions as part of Apache
> Bahir
> >> > > > project.
> >> > > >
> >> > > > Can you please share Pub/Sub streamer implementation details
with
> >> > Apache
> >> > > > Bahir community and request for feedback.
> >> > > >
> >> > > > https://bahir.apache.org/
> >> > > >
> >> > > > I have also requested for contributor access in Jira for Apache
> >> Bahir
> >> > > > project so that I can create issues and assign to myself.
> >> > > >
> >> > > > I can help with code reviews as well.
> >> > > >
> >> > > > Here is the email thread for reference
> >> > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html
> >> > > >
> >> > > > Regards,
> >> > > > Saikat
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <
> >> saikat.maitra@gmail.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hello,
> >> > > > >
> >> > > > > As discussed I have created following discussion threads
on our
> >> > > migration
> >> > > > > proposals for Apache Ignite extensions:
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html
> >> > > > >
> >> > > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html
> >> > > > >
> >> > > > > I will share update as I hear more information.
> >> > > > >
> >> > > > > Regards,
> >> > > > > Saikat
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <
> >> > saikat.maitra@gmail.com
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > >> Hello Denis,
> >> > > > >>
> >> > > > >> Sure, sounds good. I will create a separate discussion
thread
> to
> >> get
> >> > > > >> community feedback on whether we
> >> > > > >> should join the Bahir or kick-off "Ignite Extensions"
project.
> >> > > > >>
> >> > > > >> Regards,
> >> > > > >> Saikat
> >> > > > >>
> >> > > > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dmagda@apache.org
> >
> >> > > wrote:
> >> > > > >>
> >> > > > >>> Folks,
> >> > > > >>>
> >> > > > >>> The concept of Apache Bahir is precisely what we
need! Saikat,
> >> > thanks
> >> > > > for
> >> > > > >>> doing the research. Why I believe the * idea* fits
us well:
> >> > > > >>>
> >> > > > >>>    - All integrations can be stored in separate
Github
> >> repositories
> >> > > and
> >> > > > >>>    have their dev lifecycles. We've not obliged
to couple all
> >> the
> >> > > > >>> integrations
> >> > > > >>>    in a single repo. For instance, Spark can be
located in a
> >> > > dedicated
> >> > > > >>> repo
> >> > > > >>>    while streaming integrations might be in a single
one. It's
> >> up
> >> > to
> >> > > > us.
> >> > > > >>>    - All the repositories and integrations belong
to ASF.
> We're
> >> not
> >> > > > >>> dumping
> >> > > > >>>    them on Github but rather keep supporting and
developing in
> >> > > > >>> accordance with
> >> > > > >>>    ASF vision and practices.
> >> > > > >>>    - There is a way to reward contributors via committership
> and
> >> > PMC
> >> > > > >>>    membership. You won't get this if a project is
just one of
> >> the
> >> > > > >>> millions of
> >> > > > >>>    Github projects.
> >> > > > >>>
> >> > > > >>> Saikat, as Ignite PMC member, could you please kick-off
a
> >> separate
> >> > > > >>> dev-list
> >> > > > >>> discussion to involve more community members and
referring to
> >> this
> >> > > > >>> thread?
> >> > > > >>> I think the primary question the community needs
to answer
> >> whether
> >> > we
> >> > > > >>> should join the Bahir or kick-off "Ignite Extensions"
project?
> >> > > > >>>
> >> > > > >>> -
> >> > > > >>> Denis
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev
<
> >> > > > zaleslaw.sin@gmail.com>
> >> > > > >>> wrote:
> >> > > > >>>
> >> > > > >>> > Maybe we could move all our Streaming Integrations
there,
> but
> >> > what
> >> > > is
> >> > > > >>> about
> >> > > > >>> > maintaining and committer permissions to the
new
> repositories?
> >> > > > >>> > I see the list of the committers and PMC members
there
> >> > > > >>> > https://bahir.apache.org/community-members/
> >> > > > >>> >
> >> > > > >>> > Could somebody from Ignite community be added
to this list
> as
> >> a
> >> > > > >>> > PMC/committer to maintain Ignite integrations?
> >> > > > >>> > If yes, I'd happy to join this project and
collaborate with
> >> these
> >> > > > guys
> >> > > > >>> > together
> >> > > > >>> > If no, and we should start from the zero level
with external
> >> PRs
> >> > -
> >> > > > >>> hmmm,
> >> > > > >>> > it's better to have our own external repository
with
> >> ApacheBahirr
> >> > > > >>> ideology.
> >> > > > >>> >
> >> > > > >>> > Also, I agree, that all connectors could be
moved there (in
> >> > > > >>> ApacheBahirr
> >> > > > >>> > like repository), but not all modules from
the page
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >> > > > >>> > because
> >> > > > >>> > they use different parts of core modules and
could be
> >> developed
> >> > > with
> >> > > > >>> > different velocity.
> >> > > > >>> >
> >> > > > >>> > In reality, before creation of new repositories
we need wide
> >> > > > discussion
> >> > > > >>> > based on the document mentioned above.
> >> > > > >>> >
> >> > > > >>> > P.S. Of course, we could experiment in the
next release 2.8
> >> with
> >> > > one
> >> > > > >>> or two
> >> > > > >>> > integrations like twitter or ZeroMQ
> >> > > > >>> > Also, I'd like Denis Magda idea for the separate
> repositories
> >> in
> >> > > > Apache
> >> > > > >>> > Github to maintain them by anybody.
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat
Maitra <
> >> > > saikat.maitra@gmail.com
> >> > > > >:
> >> > > > >>> >
> >> > > > >>> > > Hi Denis, Emmanouil
> >> > > > >>> > >
> >> > > > >>> > > Thank you for your email. I think creating
a separate
> >> > integration
> >> > > > >>> project
> >> > > > >>> > > in github and also proposing it as an
apache incubator
> >> project
> >> > > will
> >> > > > >>> be a
> >> > > > >>> > > good move. The other separate integration
modules
> >> > > > >>> > >
> >> > > > >>> > >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >> > > > >>> > > can
> >> > > > >>> > > be moved to this new apache incubator
project.
> >> > > > >>> > >
> >> > > > >>> > > There are similar integration available
for Flink and
> Spark
> >> in
> >> > > > Apache
> >> > > > >>> > Bahir
> >> > > > >>> > > https://bahir.apache.org/
> >> > > > >>> > >
> >> > > > >>> > > Do you think we should reach out to Apache
Bahir community
> >> > with a
> >> > > > >>> > proposal
> >> > > > >>> > > or we should plan to create a separate
Apache Incubator
> >> > project?
> >> > > > >>> > >
> >> > > > >>> > > Regards,
> >> > > > >>> > > Saikat
> >> > > > >>> > >
> >> > > > >>> > >
> >> > > > >>> > >
> >> > > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil
Gkatziouras <
> >> > > > >>> > > gkatzioura@gmail.com>
> >> > > > >>> > > wrote:
> >> > > > >>> > >
> >> > > > >>> > > > Hi all!
> >> > > > >>> > > >
> >> > > > >>> > > > Based on the discussions most probably
this is going to
> >> be on
> >> > > > >>> another
> >> > > > >>> > > > GitHub repo. Therefore I will prepare
a standalone
> project
> >> > with
> >> > > > the
> >> > > > >>> > > Pub/Sub
> >> > > > >>> > > > module and once the repository is
created a pull request
> >> > shall
> >> > > be
> >> > > > >>> > issued.
> >> > > > >>> > > > If there is anything else I can do
in order to
> facilitate
> >> > this
> >> > > > >>> please
> >> > > > >>> > let
> >> > > > >>> > > > me know.
> >> > > > >>> > > >
> >> > > > >>> > > > If a ticket is created for this issue,
I shall be able
> to
> >> > > create
> >> > > > a
> >> > > > >>> pull
> >> > > > >>> > > > request and thus a build will take
place on ignite's CI.
> >> > > > >>> > > > If everyone is aligned with adding
a Pub/Sub
> >> implementation
> >> > > may I
> >> > > > >>> > create
> >> > > > >>> > > a
> >> > > > >>> > > > ticket on JIRA?
> >> > > > >>> > > >
> >> > > > >>> > > > My next question has to do on how
Ignite configures the
> >> build
> >> > > > jobs
> >> > > > >>> in
> >> > > > >>> > > Team
> >> > > > >>> > > > City. The unit tests for my implementation
need a local
> >> > Pub/Sub
> >> > > > >>> server
> >> > > > >>> > up
> >> > > > >>> > > > and running (they provide one for
testing purposes).
> Will
> >> > this
> >> > > > take
> >> > > > >>> > > effect
> >> > > > >>> > > > in a form of a build script inside
my implementation or
> >> is it
> >> > > > >>> something
> >> > > > >>> > > > that should be configured on Team
City?
> >> > > > >>> > > >
> >> > > > >>> > > > Kind regards,
> >> > > > >>> > > > Emmanouil
> >> > > > >>> > > >
> >> > > > >>> > > >
> >> > > > >>> > > > *Emmanouil Gkatziouras*
> >> > > > >>> > > > https://egkatzioura.com/ |
> >> > > > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >> > > > >>> > > > https://github.com/gkatzioura
> >> > > > >>> > > >
> >> > > > >>> > > >
> >> > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis
Magda <
> >> dmagda@apache.org
> >> > >
> >> > > > >>> wrote:
> >> > > > >>> > > >
> >> > > > >>> > > > > Emmanouil, Saikat,
> >> > > > >>> > > > >
> >> > > > >>> > > > > After contemplating on this,
let me propose another
> way
> >> > with
> >> > > > this
> >> > > > >>> > > > > particular integration that
is aligned with our
> >> > > modularization
> >> > > > >>> > > endeavor.
> >> > > > >>> > > > > The modularization [1] defines
the Ignite core, Ignite
> >> > > modules,
> >> > > > >>> and
> >> > > > >>> > > > > independent integrations. The
first two are to be
> >> located
> >> > in
> >> > > > >>> Ignite
> >> > > > >>> > > > > repositories and to be tested/updated
by the community
> >> for
> >> > > > every
> >> > > > >>> > > release.
> >> > > > >>> > > > > The modules are a subset of
existing integrations that
> >> are
> >> > > > >>> adopted
> >> > > > >>> > > widely
> >> > > > >>> > > > > and require community attention.
> >> > > > >>> > > > >
> >> > > > >>> > > > > As for the independent integrations
[1], those are
> >> supposed
> >> > > to
> >> > > > be
> >> > > > >>> > > located
> >> > > > >>> > > > > in a separate GitHub repo. I
would advise us to start
> >> > > > >>> implementing
> >> > > > >>> > this
> >> > > > >>> > > > > concept with the Pub/Sub integration.
Even though it
> >> will
> >> > be
> >> > > > >>> located
> >> > > > >>> > > in a
> >> > > > >>> > > > > separate Github repo, it will
be listed among
> officially
> >> > > > >>> available
> >> > > > >>> > > > > integrations on the Ignite website
featuring Emmanouil
> >> as a
> >> > > > >>> > > contributor.
> >> > > > >>> > > > >
> >> > > > >>> > > > > What do you think?
> >> > > > >>> > > > >
> >> > > > >>> > > > >
> >> > > > >>> > > > > [1]
> >> > > > >>> > > > >
> >> > > > >>> > > > >
> >> > > > >>> > > >
> >> > > > >>> > >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >> > > > >>> > > > >
> >> > > > >>> > > > >
> >> > > > >>> > > > >
> >> > > > >>> > > > > -
> >> > > > >>> > > > > Denis
> >> > > > >>> > > > >
> >> > > > >>> > > > >
> >> > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47
PM Saikat Maitra <
> >> > > > >>> > saikat.maitra@gmail.com
> >> > > > >>> > > >
> >> > > > >>> > > > > wrote:
> >> > > > >>> > > > >
> >> > > > >>> > > > > > Hi Emmanouil,
> >> > > > >>> > > > > >
> >> > > > >>> > > > > > The changes looks good
to me. I wanted to check if
> you
> >> > are
> >> > > > also
> >> > > > >>> > able
> >> > > > >>> > > to
> >> > > > >>> > > > > > validate that the once
data is added to Ignite
> cluster
> >> > you
> >> > > > are
> >> > > > >>> also
> >> > > > >>> > > > able
> >> > > > >>> > > > > to
> >> > > > >>> > > > > > access it using another
Ignite client or using rest
> >> > > > endpoints.
> >> > > > >>> > > > > >
> >> > > > >>> > > > > > We use Teamcity for CI
process, the details are as
> >> > follows
> >> > > > >>> > > > > >
> >> > > > >>> > > > > > Trigger validation of those
test suites that have
> been
> >> > > > >>> affected by
> >> > > > >>> > > your
> >> > > > >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>:
> <
> >> > > > >>> > > > > http://204.14.53.153/
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > >
> >> > > > >>> > > > > >    - Locate a test suite
you have to check, press
> >> button
> >> > > > named
> >> > > > >>> > "..."
> >> > > > >>> > > > that
> >> > > > >>> > > > > >    is located on the left
of "Run" button. "Run
> custom
> >> > > build"
> >> > > > >>> > dialog
> >> > > > >>> > > > will
> >> > > > >>> > > > > >    appear;
> >> > > > >>> > > > > >    - Go to "Changes" tab
and choose
> >> > > > >>> > "pull/<pull-request-number>/head"
> >> > > > >>> > > > in
> >> > > > >>> > > > > >    "Build branch" dropdown
list;
> >> > > > >>> > > > > >    - Press "Run build"
button and monitor tests
> >> results.
> >> > > > >>> > > > > >
> >> > > > >>> > > > > >
> >> > > > >>> > > > > >
> >> > > > >>> > > > > > @Roman rshtykh@yahoo.com
- can you also please
> take a
> >> > > look?
> >> > > > >>> > > > > >
> >> > > > >>> > > > > > Regards,
> >> > > > >>> > > > > > Saikat
> >> > > > >>> > > > > >
> >> > > > >>> > > > > > On Sun, Oct 13, 2019 at
6:20 PM Emmanouil
> Gkatziouras
> >> <
> >> > > > >>> > > > > > gkatzioura@gmail.com>
> >> > > > >>> > > > > > wrote:
> >> > > > >>> > > > > >
> >> > > > >>> > > > > > > Hi @Saikat Maitra
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > > > I have my implementation
on an Ignite fork on my
> >> > personal
> >> > > > >>> GitHub
> >> > > > >>> > > > > account
> >> > > > >>> > > > > > > [1]
> >> > > > >>> > > > > > > It is based on the
Kafka implementation which is
> of
> >> a
> >> > > > polling
> >> > > > >>> > > nature.
> >> > > > >>> > > > > > > GCP does not provide
a service like Kafka. Instead
> >> it
> >> > > > >>> provides us
> >> > > > >>> > > > with
> >> > > > >>> > > > > > > Pub/Sub which is just
like Kafka.
> >> > > > >>> > > > > > > Therefore using the
Kafka streamer as a guideline
> >> made
> >> > > more
> >> > > > >>> sense
> >> > > > >>> > > to
> >> > > > >>> > > > > me.
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > > > Apart from the implementation
I have also added a
> >> unit
> >> > > > test.
> >> > > > >>> > > > > > > It definitely needs
some polishing and a double
> >> check
> >> > on
> >> > > my
> >> > > > >>> side
> >> > > > >>> > to
> >> > > > >>> > > > > make
> >> > > > >>> > > > > > > the review smooth.
> >> > > > >>> > > > > > > For the tests a local
Pub/Sub [2] server has to be
> >> > setup
> >> > > > >>> however
> >> > > > >>> > I
> >> > > > >>> > > am
> >> > > > >>> > > > > not
> >> > > > >>> > > > > > > sure which is the
best way to do so with regards
> to
> >> > your
> >> > > > >>> CI/CD.
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > > > [1]
> >> > > > >>> > >
> >> > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> >> > > > >>> > > > > > > [2]
> >> > > > >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > > > *Emmanouil Gkatziouras*
> >> > > > >>> > > > > > > https://egkatzioura.com/
|
> >> > > > >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >> > > > >>> > > > > > > https://github.com/gkatzioura
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > > > On Sat, 12 Oct 2019
at 02:11, Saikat Maitra <
> >> > > > >>> > > saikat.maitra@gmail.com
> >> > > > >>> > > > >
> >> > > > >>> > > > > > > wrote:
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > > > > Hello Emmanouil,
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > > Can you please
share more info about the
> >> > > implementation?
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > > We have similar
implementation for data
> streamers
> >> > with
> >> > > > >>> Kafka
> >> > > > >>> > and
> >> > > > >>> > > > > > RocketMQ
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > >
> >> > https://apacheignite-mix.readme.io/docs/kafka-streamer
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > >
> >> > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > > Please review
and share if you have any
> questions.
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > > Regards,
> >> > > > >>> > > > > > > > Saikat
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > > On Fri, Oct 11,
2019 at 4:56 PM Emmanouil
> >> > Gkatziouras <
> >> > > > >>> > > > > > > > gkatzioura@gmail.com>
> >> > > > >>> > > > > > > > wrote:
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > > > Hi all,
> >> > > > >>> > > > > > > > >
> >> > > > >>> > > > > > > > > By mentioning
Pub/Sub I mean Google Cloud's
> >> Pub/Sub
> >> > > [1]
> >> > > > >>> > > > > > > > > Yes it is
indeed cloud specific. It is the
> >> > messaging
> >> > > > >>> service
> >> > > > >>> > > > > provided
> >> > > > >>> > > > > > > by
> >> > > > >>> > > > > > > > > GCP.
> >> > > > >>> > > > > > > > > I have made
the integration with it, but I
> >> haven't
> >> > > > >>> published
> >> > > > >>> > it
> >> > > > >>> > > > yet
> >> > > > >>> > > > > > (I
> >> > > > >>> > > > > > > am
> >> > > > >>> > > > > > > > > waiting
for the discussion/ticket and proceed
> >> with
> >> > > the
> >> > > > >>> > > > > contribution).
> >> > > > >>> > > > > > > > >
> >> > > > >>> > > > > > > > > Kind regards,
> >> > > > >>> > > > > > > > > Emmanouil
> >> > > > >>> > > > > > > > >
> >> > > > >>> > > > > > > > > [1]
> >> https://cloud.google.com/pubsub/docs/overview
> >> > > > >>> > > > > > > > >
> >> > > > >>> > > > > > > > > *Emmanouil
Gkatziouras*
> >> > > > >>> > > > > > > > > https://egkatzioura.com/
|
> >> > > > >>> > > > > > > > >
> >> https://www.linkedin.com/in/gkatziourasemmanouil/
> >> > > > >>> > > > > > > > > https://github.com/gkatzioura
> >> > > > >>> > > > > > > > >
> >> > > > >>> > > > > > > > >
> >> > > > >>> > > > > > > > > On Fri,
11 Oct 2019 at 21:45, Denis Magda <
> >> > > > >>> dmagda@apache.org
> >> > > > >>> > >
> >> > > > >>> > > > > wrote:
> >> > > > >>> > > > > > > > >
> >> > > > >>> > > > > > > > > > Hello
Emmanouil,
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > > Are
you referring to any specific
> >> implementation
> >> > > > >>> tailored
> >> > > > >>> > > for a
> >> > > > >>> > > > > > > cloud?
> >> > > > >>> > > > > > > > If
> >> > > > >>> > > > > > > > > > I'm
not mistaken you were mentioning
> AWS/GCE.
> >> Is
> >> > > > there
> >> > > > >>> any
> >> > > > >>> > > > Github
> >> > > > >>> > > > > > > > source
> >> > > > >>> > > > > > > > > > code?
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > > Here
is a list of existing integrations with
> >> > > > streaming
> >> > > > >>> and
> >> > > > >>> > > > > pub/sub
> >> > > > >>> > > > > > > > > > frameworks:
> >> > > > >>> > > > > > > > > >
> >> https://apacheignite-mix.readme.io/docs/overview
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > > with
the source code in Ignite master:
> >> > > > >>> > > > > > > > > >
> >> > > https://github.com/apache/ignite/tree/master/modules
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > > Check
out those resources to see what are
> the
> >> > most
> >> > > > >>> suitable
> >> > > > >>> > > > > Ignite
> >> > > > >>> > > > > > > APIs
> >> > > > >>> > > > > > > > > for
> >> > > > >>> > > > > > > > > > such
integrations.
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > > @Saikat
Maitra <saikat.maitra@gmail.com> ,
> >> > @Roman
> >> > > <
> >> > > > >>> > > > > > rshtykh@yahoo.com
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > > > ,
> >> > > > >>> > > > > > > > > as
> >> > > > >>> > > > > > > > > > streaming
technologies
> >> contributors/maintainers,
> >> > > > could
> >> > > > >>> you
> >> > > > >>> > > > please
> >> > > > >>> > > > > > > join
> >> > > > >>> > > > > > > > > this
> >> > > > >>> > > > > > > > > > thread?
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > > -
> >> > > > >>> > > > > > > > > > Denis
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > > On
Thu, Oct 10, 2019 at 7:58 AM Emmanouil
> >> > > > Gkatziouras <
> >> > > > >>> > > > > > > > > > gkatzioura@gmail.com>
> >> > > > >>> > > > > > > > > > wrote:
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > > > >
Hello everyone,
> >> > > > >>> > > > > > > > > > >
> >> > > > >>> > > > > > > > > > >
I started using Ignite lately. Part of my
> >> work
> >> > > > >>> involves
> >> > > > >>> > > > > Pub/Sub,
> >> > > > >>> > > > > > > > thus I
> >> > > > >>> > > > > > > > > > >
created a pub/sub streamer.
> >> > > > >>> > > > > > > > > > >
Pub/Sub is pretty close to Kafka.
> >> > > > >>> > > > > > > > > > >
Here's how the streamer works. Every node
> >> > creates
> >> > > > an
> >> > > > >>> > > instance
> >> > > > >>> > > > > of
> >> > > > >>> > > > > > a
> >> > > > >>> > > > > > > > > > Pub/Sub
> >> > > > >>> > > > > > > > > > >
client.
> >> > > > >>> > > > > > > > > > >
The clients subscribe to the same topic
> >> using
> >> > the
> >> > > > >>> same
> >> > > > >>> > > > > > subscription
> >> > > > >>> > > > > > > > > name.
> >> > > > >>> > > > > > > > > > >
A message is being sent and one of the
> nodes
> >> > will
> >> > > > >>> receive
> >> > > > >>> > > the
> >> > > > >>> > > > > > > > message,
> >> > > > >>> > > > > > > > > > >
process it and then forward it to Ignite
> >> using
> >> > > the
> >> > > > >>> > > extractor
> >> > > > >>> > > > > > > > provided.
> >> > > > >>> > > > > > > > > > >
> >> > > > >>> > > > > > > > > > >
Any thoughts?
> >> > > > >>> > > > > > > > > > >
> >> > > > >>> > > > > > > > > > >
*Emmanouil Gkatziouras*
> >> > > > >>> > > > > > > > > > >
https://egkatzioura.com/ |
> >> > > > >>> > > > > > > > > > >
> >> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >> > > > >>> > > > > > > > > > >
https://github.com/gkatzioura
> >> > > > >>> > > > > > > > > > >
> >> > > > >>> > > > > > > > > >
> >> > > > >>> > > > > > > > >
> >> > > > >>> > > > > > > >
> >> > > > >>> > > > > > >
> >> > > > >>> > > > > >
> >> > > > >>> > > > >
> >> > > > >>> > > >
> >> > > > >>> > >
> >> > > > >>> >
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message