kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Replay
Date Thu, 29 Mar 2012 17:10:32 GMT
Elben,

autooffset.reset="largest" is only used for the very first time the
consumer is started when there is no offset registered in ZK. Once the
consumer is started, by default it periodically checkpoints offset in ZK
and on subsequent restarts, the consumer starts consumption from the offset
stored in ZK.

Thanks,

Jun

On Thu, Mar 29, 2012 at 9:54 AM, Elben Shira <elbenshira@gmail.com> wrote:

> Hi Neha,
>
> What does it mean to fetch the latest data? If there are items in the kafka
> queue and no new items are coming in, would autooffset.reset="largest" wait
> for new items or would it consume some of the existing items in the queue?
>
> I'm asking this because I've modified the sensei gateways
> to autooffset.reset="largest" but it's still consuming items previously in
> the queue, and I'm not sure if the problem is on my end.
>
> I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find the
> latest offset for the queue itself (not the consumer)?
>
> Elben
>
>
> On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > John,
> >
> > You can achieve that via the autooffset.reset="largest" config option
> > in the consumer.
> >
> > Thanks,
> > Neha
> >
> > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <john.wang@gmail.com> wrote:
> > > Hi Neha:
> > >
> > >     Can you point me to the api to always get data from tail?
> > >
> > > Thanks
> > >
> > > -John
> > >
> > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > >wrote:
> > >
> > >> Elben,
> > >>
> > >> Or sensei can provide a restart from tail API, that will delete the
> > offset
> > >> state from ZK and restart its Kafka consumer. Currently that is the
> best
> > >> way to have a Kafka consumer restart fetching latest data.
> > >>
> > >> Thanks,
> > >> Neha
> > >> On Mar 22, 2012 4:39 PM, "John Wang" <john.wang@gmail.com> wrote:
> > >>
> > >> > I think in the newer kafka release, there is a tool to update the
> > version
> > >> > for a client.
> > >> >
> > >> > Jun, can you elaborate on that?
> > >> >
> > >> > Thanks
> > >> >
> > >> > -John
> > >> >
> > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <elbenshira@gmail.com>
> > >> wrote:
> > >> >
> > >> > > We're using use kafka+sensei at our company and we'd love to
be
> > able to
> > >> > > start at the end of the queue (latest entries). The use case
is
> when
> > >> > sensei
> > >> > > goes down or has to be restarted, we only care about indexing
the
> > >> latest
> > >> > > items.
> > >> > >
> > >> > > A workaround I guess is to stop sensei, manually change the high
> > >> > watermark
> > >> > > in kafka, then restart sensei.
> > >> > >
> > >> > > Elben
> > >> > >
> > >> > >
> > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <junrao@gmail.com>
> wrote:
> > >> > >
> > >> > > > Casey,
> > >> > > >
> > >> > > > You can't get offsets using zk-based consumer now. We are
still
> > >> trying
> > >> > to
> > >> > > > figure about how to support that in the future.
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
> > >> > > > Casey.Sybrandy@six3systems.com> wrote:
> > >> > > >
> > >> > > > > Hello,
> > >> > > > >
> > >> > > > > Just out of curiosity, how would you get the offsets
when
> using
> > a
> > >> > > > zk-based
> > >> > > > > consumer?  I don't see anything that seems to let you
do this
> > >> easily.
> > >> > > > >
> > >> > > > > Casey
> > >> > > > >
> > >> > > > > ________________________________________
> > >> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
> > >> > > > > Sent: Monday, March 12, 2012 5:51 PM
> > >> > > > > To: kafka-users@incubator.apache.org
> > >> > > > > Subject: Re: Replay
> > >> > > > >
> > >> > > > > With simple consumer, you can start from any valid
offset that
> > you
> > >> > keep
> > >> > > > > track off.
> > >> > > > >
> > >> > > > > With the high level (zk-based) consumer: you will need
to stop
> > all
> > >> > > > > consumers, and then update the offsets in zookeeper
(see the
> > >> > > > > ImportZkOffsets tool in the tools package) with the
offsets
> you
> > >> want
> > >> > to
> > >> > > > > replay from before restarting your consumer.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > >
> > >> > > > > Joel
> > >> > > > >
> > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <
> > JainK@bit-sys.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Hi,
> > >> > > > > >
> > >> > > > > > Sorry, new to Kafka. Is it possible for a consumer
to replay
> > >> > messages
> > >> > > > > from
> > >> > > > > > an "offset" that may have been committed before?
> > >> > > > > >
> > >> > > > > > Thanks for the help,
> > >> > > > > > Kokil Jain
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message