kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From noah <iamn...@gmail.com>
Subject Re: kafka consumer group API
Date Thu, 09 Jul 2015 22:11:05 GMT
Hi!

I did something similar. You can use the high level consumer but turn off
auto commit and commit only what you are done with. Here's the code I used:
https://github.com/iamnoah/kakfa-offsets-test

On Thu, Jul 9, 2015 at 4:53 PM Shashank Singh <shashank.rudra@gmail.com>
wrote:

> Hi Team
>
> I was going over the documentation and links related to consumer API
> provided in
> 0.8.2.1 release
> I got to know of major improvement in consumer API in 0.8.3 release. May I
> know when we are planning to have this release.
>
> One more help - I was following up the example given on
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
>
> In this approach we can't decide
> a) how to get the current offset
> b) how to set a particular consumer group to start reading from a specific
> offset
>
> Also how to use the same example to read one message at a time from a topic
> with many topics. I tried with creating a list of iterators and then
> checking for hasNext() but that call in blocking. Even though I am using a
> timeout on that call this seems not be working.
>
> Instead what I have settled with is an internal queue to which the
> multi-threaded consumer offers the messages into. Poll API polls the
> internal queue. Problem is in case client code gets down without consuming
> all the messages the offset is already set and the messages which have
> already been read from iterator and put in queue will be never seen again.
>
>
> --
>
> *Warm Regards,*
>
> *Shashank  *
>
> *Mobile: +91 9910478553 *
>
> *Linkedin: in.linkedin.com/pub/shashank-singh/13/763/906/
> <http://in.linkedin.com/pub/shashank-singh/13/763/906/>*
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message