kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradeep Bhattiprolu <pbhatt...@gmail.com>
Subject Re: Kafka Newbie question
Date Sun, 10 Apr 2016 04:48:53 GMT
Liquan , thanks for the response.
By setting the auto commit to false do i have to manage queue offset
manually ?
I am running a multiple threads with each thread being a consumer, it would
be complicated to manage offsets across threads, if i dont use kafka's
automatic consumer group abstraction.

Thanks
Pradeep

On Sat, Apr 9, 2016 at 3:12 AM, Liquan Pei <liquanpei@gmail.com> wrote:

> Hi Pradeep,
>
> Can you try to set enable.auto.commit = false if you want to read to the
> earliest offset? According to the documentation, auto.offset.reset controls
> what to do when there is no initial offset in Kafka or if the current
> offset does not exist any more on the server (e.g. because that data has
> been deleted). In case that auto commit is enabled, the committed offset is
> available in some servers.
>
> Thanks,
> Liquan
>
> On Fri, Apr 8, 2016 at 10:44 PM, Pradeep Bhattiprolu <pbhattip9@gmail.com>
> wrote:
>
> > Hi All
> >
> > I am a newbie to kafka. I am using the new Consumer API in a thread
> acting
> > as a consumer for a topic in Kafka.
> > For my testing and other purposes I have read the queue multiple times
> > using console-consumer.sh script of kafka.
> >
> > To start reading the message from the beginning in my java code , I have
> > set the value of the auto.offset.reset to "earliest".
> >
> > However that property does not guarantee that i begin reading the
> messages
> > from start, it goes by the most recent smallest offset for the consumer
> > group.
> >
> > Here is my question,
> > Is there a assured way of starting to read the messages from beginning
> from
> > Java based Kafka Consumer ?
> > Once I reset one of my consumers to zero, do i have to do offset
> management
> > myself for other consumer threads or does kafka automatically lower the
> > offset to the first threads read offset ?
> >
> > Any information / material pointing to the solution are highly
> appreciated.
> >
> > Thanks
> > Pradeep
> >
>
>
>
> --
> Liquan Pei
> Software Engineer, Confluent Inc
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message