kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: Consumer re-design proposal
Date Mon, 18 Jun 2012 20:41:33 GMT
That's true - I think that's one of the major motivations of the consumer
re-design. Right now, the consumer implementation is very thick which makes
it difficult to maintain correct implementations across multiple languages.
It will be much easier to implement a consumer with the thinner logic - and
as you pointed out, many languages have pretty good bindings with native C
libraries so technically we would go pretty far with just a JVM and native
(C) implementation of the consumer logic.

Joel

On Mon, Jun 18, 2012 at 11:40 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:

> Would porting the consumer/producer code to C be a good idea?  I say this
> because at least with most languages I know of, leveraging a C library is
> pretty easy.  This way, you would have to maintain only the C library and
> others can make/maintain wrappers for their languages.  Having to port to
> other languages is going to cause you to have a significant amount of
> maintenance if you change the protocol in the future.
>
> ________________________________________
> From: Neha Narkhede [neha.narkhede@gmail.com]
> Sent: Thursday, June 14, 2012 5:53 PM
> To: kafka-users@incubator.apache.org
> Cc: kafka-dev@incubator.apache.org
> Subject: Re: Consumer re-design proposal
>
> Thanks for the feedback ! I moved it to
> https://issues.apache.org/jira/browse/KAFKA-364, so that we can keep track
> of these.
>
> -Neha
>
> On Thu, Jun 14, 2012 at 2:45 PM, Marcos Juarez <mjuarez@gmail.com> wrote:
>
> > Throwing a +1 on "Allow the consumer to reset its offset to some
> arbitrary
> > value, and then write that offset into ZK".
> >
> > We're currently running into a scenario where we would like to have 100%
> > reliability, and we're losing a few messages when a connection is broken,
> > but there were still a few messages in the OS TCP buffers. So, we're
> > planning on shifting the ZK offset by a few seconds "back in time" if we
> > detect a broker has gone down, to make sure all the messages will be
> > actually delivered to the end consumer when that broker comes back up,
> even
> > if there's a small amount of overlapping messages.
> >
> > Thanks,
> >
> > Marcos
> >
> >
> > On Jun 14, 2012, at 2:39 PM, Evan Chan wrote:
> >
> > > I would like to throw in a couple use cases:
> > >
> > >
> > >   - Allow the new consumer to reset its offset to either the current
> > >   largest or smallest.  This would be a great way to restart a process
> > that
> > >   has fallen behind.  The only way I know how to do this today, with
> the
> > >   high-level consumer, is to delete the ZK nodes manually and restart
> the
> > >   consumer.
> > >   - Allow the consumer to reset its offset to some arbitrary value, and
> > >   then write that offset into ZK.    Kind of like the first case, but
> > would
> > >   make rewinding/replays much easier.
> > >
> > > Modularity (the ability to layer the ZK infrastructure on top of the
> > simple
> > > interface) would be great.
> > >
> > > thanks,
> > > Evan
> > >
> > >
> > > On Tue, Jun 12, 2012 at 9:59 AM, Jay Kreps <jay.kreps@gmail.com>
> wrote:
> > >
> > >> This is a great summary Neha. It would be good to get people's
> feedback
> > on
> > >> this since we don't want to keep breaking api and
> > >> protocol compatibility here, so the hope is to really get it right
> this
> > >> time now that we have really seen all the use cases and live with the
> > >> output for a while. I think the consumer design is a pretty hard
> > protocol
> > >> and API design problem, so its fun to think about.
> > >>
> > >> If I were to summarize Neha's requirements list, I think there are
> three
> > >> high-level goals:
> > >>
> > >>  1. Simplify the consumer protocol to enable ease of development of
> > >>  consumer clients in other languages
> > >>  2. Try to replace the "simple consumer" and "high level consumer"
> with
> > a
> > >>  single, general interface that has all the advantages of both.
> > >>  3. Support a bunch of use cases that either we didn't think of, or
> that
> > >>  weren't possible in the partitioning model of the pre-0.8 code base.
> > >>
> > >> -Jay
> > >>
> > >>
> > >> On Mon, Jun 11, 2012 at 4:52 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > >>> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Over the past few months, we've received quite a lot of feedback on
> the
> > >>> consumer side features and design. Some of them are improvements to
> the
> > >>> current consumer design and some are simply new feature/API
> requests. I
> > >>> have attempted to write up the requirements that I've heard on this
> > wiki
> > >> -
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > >>>
> > >>> This would involve some significant changes to the consumer APIs, so
> we
> > >>> would like to collect feedback on the proposal from our community.
> > Since
> > >>> the list of changes is not small, we would like to understand if some
> > >>> features are preferred over others, and more importantly, if some
> > >> features
> > >>> are not required at all.
> > >>>
> > >>> Since some part of this proposal is experimental and the consumer
> side
> > >>> changes are non-trivial, we would like this initiative to not
> interfere
> > >>> with the forthcoming replication release. However, it will be good
to
> > >> have
> > >>> people from the community give this some thought and help out with
> the
> > >>> JIRAs if interested. One way of managing this project could be
> > creating a
> > >>> separate branch from the kafka trunk and continue development on it.
> > Once
> > >>> it is ready and in good shape, we can think about cutting another
> > release
> > >>> (after 0.8) for the releasing the new consumer API. Do people have
> > >>> preferences/concerns regarding creating a separate branch for this
> > >> project
> > >>> ?
> > >>>
> > >>> Please feel free to start a discussion on this JIRA -
> > >>> https://issues.apache.org/jira/browse/KAFKA-364
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Neha
> > >>>
> > >>
> > >
> > >
> > >
> > > --
> > > --
> > > *Evan Chan*
> > > Senior Software Engineer |
> > > ev@ooyala.com | (650) 996-4600
> > > www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > > @ooyala<http://www.twitter.com/ooyala>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message