kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrienne Kole <adrienneko...@gmail.com>
Subject Re: Plans to extend streams?
Date Wed, 29 Nov 2017 15:15:30 GMT
Not necessarily.

I would avoid the term "reimplemented". Btw, Apache Storm is not also the
best (streaming) system that can utilize the network and it does not
support runtime scale in/out (at least by design).
So, can streams preserve its current selling points (ex:dynamicity) while
introducing a new communication protocol, which will include the current
one (streams-kafka-streams) and the new one (streams-streams)? Are there
plans over this?



On Wed, Nov 29, 2017 at 3:56 PM, Wim Van Leuven <
wim.vanleuven@highestpoint.biz> wrote:

> What you are actually asking is if Kafka Streams should be reimplemented as
> Apache Storm?
> -wim
>
> On Wed, 29 Nov 2017 at 15:10 Adrienne Kole <adriennekole1@gmail.com>
> wrote:
>
> > Hi,
> >
> > The purpose of this email is to get overall intuition for the future
> plans
> > of streams library.
> >
> > The main question is that, will it be a single threaded application in
> the
> > long run and serve microservices use-cases, or are there any plans to
> > extend it to multi-node execution framework with less kafka dependency.
> >
> > Currently, each streams node 'talks' with kafka cluster and they can
> > indirectly talk with each other again through kafka. However, especially
> if
> > kafka is not in the same network with streams nodes (actually this can
> > happen if they are in the same network as well) this will cause high
> > network overhead and inefficiency.
> >
> > One solution for this (bypassing network overhead) is to deploy streams
> > node on kafka cluster to ensure the data locality. However, this is not
> > recommended as the library and kafka can affect each other's performance
> > and  streams does not necessarily have to know the internal data
> > partitioning of kafka.
> >
> > Another solution would be extending streams library to have a common
> > runtime. IMO, preserving the current selling points of streams (like
> > dynamic scale in/out) with this kind of extensions can be very good
> > improvement.
> >
> > So my question is that, will streams in the long/short run, will extend
> its
> > use-cases to massive and efficient stream processing (and compete with
> > spark) or stay and strengthen its current position?
> >
> > Cheers,
> > Adrienne
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message