kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Binita Bharati <binita.bhar...@gmail.com>
Subject Re: Consumer group ID for high level consumer
Date Tue, 04 Mar 2014 04:47:08 GMT
Hi Neha,


I had to set the "auto.offset.reset" to "smallest" in the particular case
where consumer threads weren't running when the data was produced.

Then on, its working as you said, ie , as long as the group.id is same, the
consumer threads continue to receive new data.

Thanks


On Mon, Mar 3, 2014 at 10:55 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Question  : How to make the consumer threads receive the older data that
> was produced when the consumer threads aren't running ?
>
> As long as you make sure the group.id doesn't change, the consumer will
> always
> consume data from where it left off during its last run. In other words,
> the consumer
> can consume data that was produced when it wasn't running, if it is always
> started
> using the same group.id
>
> Thanks,
> Neha
>
>
> On Mon, Mar 3, 2014 at 8:02 AM, Binita Bharati <binita.bharati@gmail.com
> >wrote:
>
> > Hi all,
> >
> > The HighLevelConsumer is working fine. Ref :
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> .
> >
> > I was expecting the consumer threads to be able to receive older data (
> ie
> > the data put into the topic when the consumer threads weren't running )
> > That was the root of the problem, and once, I started my consumer
> > application first, and then, generated fresh data into the topic, the
> > consumer threads started receiving the new data.
> >
> > Question  : How to make the consumer threads receive the older data that
> > was produced when the consumer threads aren't running ?
> >
> > Thanks
> >
> >
> > On Wed, Feb 26, 2014 at 6:30 PM, Martin Kleppmann
> > <mkleppmann@linkedin.com>wrote:
> >
> > > Hi Binita,
> > >
> > > The consumer group (group.id) is a mechanism for sharing the load of
> > > consuming a high-volume topic between multiple consumers. If you don't
> > set
> > > a group ID, each consumer consumes all the partitions of a topic. If
> you
> > > set several consumers to the same group ID, the partitions of the topic
> > > you're consuming will be shared amongst the consumers in that group, so
> > > that each message is delivered to only one of the consumers in the
> group.
> > >
> > > You can set the group ID to be whatever you like.
> > >
> > > Hope that helps,
> > > Martin
> > >
> > > On 25 Feb 2014, at 06:27, Binita Bharati <binita.bharati@gmail.com>
> > wrote:
> > > > Hi all,
> > > >
> > > > I am referring to this e.g:
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > .
> > > >
> > > > What is the consumer group ID being referred here ?
> > > >
> > > > Thanks
> > > > Binita
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message