kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Consumer re-design proposal
Date Mon, 18 Jun 2012 22:29:52 GMT
This would definitely be good to have. As Joel says we are thinking
about re-factoring the consumer protocol to make it much thinner to help
enable non-java consumers much more easily (i.e. get rid of zookeeper and
other things).

There is a very brief write up on this here:
https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination

Take a look and let us know what you think. It might make sense to wait
until this is done before adding consumer clients.

-Jay

On Mon, Jun 18, 2012 at 1:41 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> That's true - I think that's one of the major motivations of the consumer
> re-design. Right now, the consumer implementation is very thick which makes
> it difficult to maintain correct implementations across multiple languages.
> It will be much easier to implement a consumer with the thinner logic - and
> as you pointed out, many languages have pretty good bindings with native C
> libraries so technically we would go pretty far with just a JVM and native
> (C) implementation of the consumer logic.
>
> Joel
>
> On Mon, Jun 18, 2012 at 11:40 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Would porting the consumer/producer code to C be a good idea?  I say this
> > because at least with most languages I know of, leveraging a C library is
> > pretty easy.  This way, you would have to maintain only the C library and
> > others can make/maintain wrappers for their languages.  Having to port to
> > other languages is going to cause you to have a significant amount of
> > maintenance if you change the protocol in the future.
> >
> > ________________________________________
> > From: Neha Narkhede [neha.narkhede@gmail.com]
> > Sent: Thursday, June 14, 2012 5:53 PM
> > To: kafka-users@incubator.apache.org
> > Cc: kafka-dev@incubator.apache.org
> > Subject: Re: Consumer re-design proposal
> >
> > Thanks for the feedback ! I moved it to
> > https://issues.apache.org/jira/browse/KAFKA-364, so that we can keep
> track
> > of these.
> >
> > -Neha
> >
> > On Thu, Jun 14, 2012 at 2:45 PM, Marcos Juarez <mjuarez@gmail.com>
> wrote:
> >
> > > Throwing a +1 on "Allow the consumer to reset its offset to some
> > arbitrary
> > > value, and then write that offset into ZK".
> > >
> > > We're currently running into a scenario where we would like to have
> 100%
> > > reliability, and we're losing a few messages when a connection is
> broken,
> > > but there were still a few messages in the OS TCP buffers. So, we're
> > > planning on shifting the ZK offset by a few seconds "back in time" if
> we
> > > detect a broker has gone down, to make sure all the messages will be
> > > actually delivered to the end consumer when that broker comes back up,
> > even
> > > if there's a small amount of overlapping messages.
> > >
> > > Thanks,
> > >
> > > Marcos
> > >
> > >
> > > On Jun 14, 2012, at 2:39 PM, Evan Chan wrote:
> > >
> > > > I would like to throw in a couple use cases:
> > > >
> > > >
> > > >   - Allow the new consumer to reset its offset to either the current
> > > >   largest or smallest.  This would be a great way to restart a
> process
> > > that
> > > >   has fallen behind.  The only way I know how to do this today, with
> > the
> > > >   high-level consumer, is to delete the ZK nodes manually and restart
> > the
> > > >   consumer.
> > > >   - Allow the consumer to reset its offset to some arbitrary value,
> and
> > > >   then write that offset into ZK.    Kind of like the first case, but
> > > would
> > > >   make rewinding/replays much easier.
> > > >
> > > > Modularity (the ability to layer the ZK infrastructure on top of the
> > > simple
> > > > interface) would be great.
> > > >
> > > > thanks,
> > > > Evan
> > > >
> > > >
> > > > On Tue, Jun 12, 2012 at 9:59 AM, Jay Kreps <jay.kreps@gmail.com>
> > wrote:
> > > >
> > > >> This is a great summary Neha. It would be good to get people's
> > feedback
> > > on
> > > >> this since we don't want to keep breaking api and
> > > >> protocol compatibility here, so the hope is to really get it right
> > this
> > > >> time now that we have really seen all the use cases and live with
> the
> > > >> output for a while. I think the consumer design is a pretty hard
> > > protocol
> > > >> and API design problem, so its fun to think about.
> > > >>
> > > >> If I were to summarize Neha's requirements list, I think there are
> > three
> > > >> high-level goals:
> > > >>
> > > >>  1. Simplify the consumer protocol to enable ease of development of
> > > >>  consumer clients in other languages
> > > >>  2. Try to replace the "simple consumer" and "high level consumer"
> > with
> > > a
> > > >>  single, general interface that has all the advantages of both.
> > > >>  3. Support a bunch of use cases that either we didn't think of, or
> > that
> > > >>  weren't possible in the partitioning model of the pre-0.8 code
> base.
> > > >>
> > > >> -Jay
> > > >>
> > > >>
> > > >> On Mon, Jun 11, 2012 at 4:52 PM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > > >>> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> Over the past few months, we've received quite a lot of feedback
on
> > the
> > > >>> consumer side features and design. Some of them are improvements
to
> > the
> > > >>> current consumer design and some are simply new feature/API
> > requests. I
> > > >>> have attempted to write up the requirements that I've heard on
this
> > > wiki
> > > >> -
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > >>>
> > > >>> This would involve some significant changes to the consumer APIs,
> so
> > we
> > > >>> would like to collect feedback on the proposal from our community.
> > > Since
> > > >>> the list of changes is not small, we would like to understand
if
> some
> > > >>> features are preferred over others, and more importantly, if some
> > > >> features
> > > >>> are not required at all.
> > > >>>
> > > >>> Since some part of this proposal is experimental and the consumer
> > side
> > > >>> changes are non-trivial, we would like this initiative to not
> > interfere
> > > >>> with the forthcoming replication release. However, it will be
good
> to
> > > >> have
> > > >>> people from the community give this some thought and help out
with
> > the
> > > >>> JIRAs if interested. One way of managing this project could be
> > > creating a
> > > >>> separate branch from the kafka trunk and continue development
on
> it.
> > > Once
> > > >>> it is ready and in good shape, we can think about cutting another
> > > release
> > > >>> (after 0.8) for the releasing the new consumer API. Do people
have
> > > >>> preferences/concerns regarding creating a separate branch for
this
> > > >> project
> > > >>> ?
> > > >>>
> > > >>> Please feel free to start a discussion on this JIRA -
> > > >>> https://issues.apache.org/jira/browse/KAFKA-364
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Neha
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > --
> > > > *Evan Chan*
> > > > Senior Software Engineer |
> > > > ev@ooyala.com | (650) 996-4600
> > > > www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > > > @ooyala<http://www.twitter.com/ooyala>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message