kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From noah <iamn...@gmail.com>
Subject Re: kafka consumer group API
Date Thu, 09 Jul 2015 22:11:05 GMT

I did something similar. You can use the high level consumer but turn off
auto commit and commit only what you are done with. Here's the code I used:

On Thu, Jul 9, 2015 at 4:53 PM Shashank Singh <shashank.rudra@gmail.com>

> Hi Team
> I was going over the documentation and links related to consumer API
> provided in
> release
> I got to know of major improvement in consumer API in 0.8.3 release. May I
> know when we are planning to have this release.
> One more help - I was following up the example given on
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> In this approach we can't decide
> a) how to get the current offset
> b) how to set a particular consumer group to start reading from a specific
> offset
> Also how to use the same example to read one message at a time from a topic
> with many topics. I tried with creating a list of iterators and then
> checking for hasNext() but that call in blocking. Even though I am using a
> timeout on that call this seems not be working.
> Instead what I have settled with is an internal queue to which the
> multi-threaded consumer offers the messages into. Poll API polls the
> internal queue. Problem is in case client code gets down without consuming
> all the messages the offset is already set and the messages which have
> already been read from iterator and put in queue will be never seen again.
> --
> *Warm Regards,*
> *Shashank  *
> *Mobile: +91 9910478553 *
> *Linkedin: in.linkedin.com/pub/shashank-singh/13/763/906/
> <http://in.linkedin.com/pub/shashank-singh/13/763/906/>*

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message