kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Replay
Date Thu, 29 Mar 2012 17:31:35 GMT
There is a tool ImportZkOffsets that allows you to import offsets (assuming
they are valid) to ZK. However, it's an admin tool and you need to stop the
consumer before using the tool.

Thanks,

Jun

On Thu, Mar 29, 2012 at 10:16 AM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Elben,
>
> What I had suggested was the following -
>
> >> Or sensei can provide a restart from tail API, that will delete the
> offset state from ZK and restart its Kafka consumer. Currently that is the
> best way to have a Kafka consumer restart fetching latest data.
>
> Note that sensei would need to first get rid of the current state for
> the consumer group from ZK (delete /consumers/[your-consumer-group]).
> And then when you restart the Kafka consumer, it will start consuming
> from the latest data available on the server.
>
> Thanks,
> Neha
>
> On Thu, Mar 29, 2012 at 10:10 AM, Jun Rao <junrao@gmail.com> wrote:
> > Elben,
> >
> > autooffset.reset="largest" is only used for the very first time the
> > consumer is started when there is no offset registered in ZK. Once the
> > consumer is started, by default it periodically checkpoints offset in ZK
> > and on subsequent restarts, the consumer starts consumption from the
> offset
> > stored in ZK.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 29, 2012 at 9:54 AM, Elben Shira <elbenshira@gmail.com>
> wrote:
> >
> >> Hi Neha,
> >>
> >> What does it mean to fetch the latest data? If there are items in the
> kafka
> >> queue and no new items are coming in, would autooffset.reset="largest"
> wait
> >> for new items or would it consume some of the existing items in the
> queue?
> >>
> >> I'm asking this because I've modified the sensei gateways
> >> to autooffset.reset="largest" but it's still consuming items previously
> in
> >> the queue, and I'm not sure if the problem is on my end.
> >>
> >> I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find
> the
> >> latest offset for the queue itself (not the consumer)?
> >>
> >> Elben
> >>
> >>
> >> On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkhede@gmail.com
> >> >wrote:
> >>
> >> > John,
> >> >
> >> > You can achieve that via the autooffset.reset="largest" config option
> >> > in the consumer.
> >> >
> >> > Thanks,
> >> > Neha
> >> >
> >> > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <john.wang@gmail.com>
> wrote:
> >> > > Hi Neha:
> >> > >
> >> > >     Can you point me to the api to always get data from tail?
> >> > >
> >> > > Thanks
> >> > >
> >> > > -John
> >> > >
> >> > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <
> >> neha.narkhede@gmail.com
> >> > >wrote:
> >> > >
> >> > >> Elben,
> >> > >>
> >> > >> Or sensei can provide a restart from tail API, that will delete
the
> >> > offset
> >> > >> state from ZK and restart its Kafka consumer. Currently that is
the
> >> best
> >> > >> way to have a Kafka consumer restart fetching latest data.
> >> > >>
> >> > >> Thanks,
> >> > >> Neha
> >> > >> On Mar 22, 2012 4:39 PM, "John Wang" <john.wang@gmail.com>
wrote:
> >> > >>
> >> > >> > I think in the newer kafka release, there is a tool to update
the
> >> > version
> >> > >> > for a client.
> >> > >> >
> >> > >> > Jun, can you elaborate on that?
> >> > >> >
> >> > >> > Thanks
> >> > >> >
> >> > >> > -John
> >> > >> >
> >> > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <
> elbenshira@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > We're using use kafka+sensei at our company and we'd
love to be
> >> > able to
> >> > >> > > start at the end of the queue (latest entries). The
use case is
> >> when
> >> > >> > sensei
> >> > >> > > goes down or has to be restarted, we only care about
indexing
> the
> >> > >> latest
> >> > >> > > items.
> >> > >> > >
> >> > >> > > A workaround I guess is to stop sensei, manually change
the
> high
> >> > >> > watermark
> >> > >> > > in kafka, then restart sensei.
> >> > >> > >
> >> > >> > > Elben
> >> > >> > >
> >> > >> > >
> >> > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <junrao@gmail.com>
> >> wrote:
> >> > >> > >
> >> > >> > > > Casey,
> >> > >> > > >
> >> > >> > > > You can't get offsets using zk-based consumer now.
We are
> still
> >> > >> trying
> >> > >> > to
> >> > >> > > > figure about how to support that in the future.
> >> > >> > > >
> >> > >> > > > Thanks,
> >> > >> > > >
> >> > >> > > > Jun
> >> > >> > > >
> >> > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey
<
> >> > >> > > > Casey.Sybrandy@six3systems.com> wrote:
> >> > >> > > >
> >> > >> > > > > Hello,
> >> > >> > > > >
> >> > >> > > > > Just out of curiosity, how would you get the
offsets when
> >> using
> >> > a
> >> > >> > > > zk-based
> >> > >> > > > > consumer?  I don't see anything that seems
to let you do
> this
> >> > >> easily.
> >> > >> > > > >
> >> > >> > > > > Casey
> >> > >> > > > >
> >> > >> > > > > ________________________________________
> >> > >> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
> >> > >> > > > > Sent: Monday, March 12, 2012 5:51 PM
> >> > >> > > > > To: kafka-users@incubator.apache.org
> >> > >> > > > > Subject: Re: Replay
> >> > >> > > > >
> >> > >> > > > > With simple consumer, you can start from any
valid offset
> that
> >> > you
> >> > >> > keep
> >> > >> > > > > track off.
> >> > >> > > > >
> >> > >> > > > > With the high level (zk-based) consumer: you
will need to
> stop
> >> > all
> >> > >> > > > > consumers, and then update the offsets in
zookeeper (see
> the
> >> > >> > > > > ImportZkOffsets tool in the tools package)
with the offsets
> >> you
> >> > >> want
> >> > >> > to
> >> > >> > > > > replay from before restarting your consumer.
> >> > >> > > > >
> >> > >> > > > > Thanks,
> >> > >> > > > >
> >> > >> > > > > Joel
> >> > >> > > > >
> >> > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil
<
> >> > JainK@bit-sys.com>
> >> > >> > > wrote:
> >> > >> > > > >
> >> > >> > > > > > Hi,
> >> > >> > > > > >
> >> > >> > > > > > Sorry, new to Kafka. Is it possible for
a consumer to
> replay
> >> > >> > messages
> >> > >> > > > > from
> >> > >> > > > > > an "offset" that may have been committed
before?
> >> > >> > > > > >
> >> > >> > > > > > Thanks for the help,
> >> > >> > > > > > Kokil Jain
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message