kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim <jimi...@gmail.com>
Subject Re: Most common kafka client comsumer implementations?
Date Fri, 01 Aug 2014 15:05:34 GMT
Thanks Guozhang,

I was looking for actual real world workflows. I realize you can commit
after each message but if you’re using ZK for offsets for instance you’ll
put too much write load on the nodes and crush your throughput. So I was
interested in batching strategies people have used that balance high/full
throughput and fully committed events.


On Thu, Jul 31, 2014 at 8:16 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Hi Jim,
>
> Whether to use high level or simple consumer depends on your use case. If
> you need to manually manage partition assignments among your consumers, or
> you need to commit your offsets elsewhere than ZK, or you do not want auto
> rebalancing of consumers upon failures etc, you will use simple consumers;
> otherwise you use high level consumer.
>
> From your description of pulling a batch of messages it seems you are
> currently using the simple consumer. Suppose you are using the high level
> consumer, to achieve at-lease-once basically you can do sth like:
>
> message = consumer.iter.next()
> process(message)
> consumer.commit()
>
> which is effectively the same as option 2 for using a simple consumer. Of
> course, doing so has a heavy overhead of one-commit-per-message, you can
> also do option 1, by the cost of duplicates, which is tolerable for
> at-least-once.
>
> Guozhang
>
>
> On Wed, Jul 30, 2014 at 8:25 PM, Jim <jiminoc@gmail.com> wrote:
>
> > Curious on a couple questions...
> >
> > Are most people(are you?) using the simple consumer vs the high level
> > consumer in production?
> >
> >
> > What is the common processing paradigm for maintaining a full pipeline
> for
> > kafka consumers for at-least-once messaging? E.g. you pull a batch of
> 1000
> > messages and:
> >
> > option 1.
> > you wait for the slowest worker to finish working on that message, when
> you
> > get back 1000 acks internally you commit your offset and pull another
> batch
> >
> > option 2.
> > you feed your workers n msgs at a time in sequence and move your offset
> up
> > as you work through your batch
> >
> > option 3.
> > you maintain a full stream of 1000 messages ideally and as you get acks
> > back from your workers you see if you can move your offset up in the
> stream
> > to pull n more messages to fill up your pipeline so you're not blocked by
> > the slowest consumer (probability wise)
> >
> >
> > any good docs or articles on the subject would be great, thanks!
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message