kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Replay
Date Thu, 29 Mar 2012 17:16:56 GMT
Elben,

What I had suggested was the following -

>> Or sensei can provide a restart from tail API, that will delete the offset state
from ZK and restart its Kafka consumer. Currently that is the best way to have a Kafka consumer
restart fetching latest data.

Note that sensei would need to first get rid of the current state for
the consumer group from ZK (delete /consumers/[your-consumer-group]).
And then when you restart the Kafka consumer, it will start consuming
from the latest data available on the server.

Thanks,
Neha

On Thu, Mar 29, 2012 at 10:10 AM, Jun Rao <junrao@gmail.com> wrote:
> Elben,
>
> autooffset.reset="largest" is only used for the very first time the
> consumer is started when there is no offset registered in ZK. Once the
> consumer is started, by default it periodically checkpoints offset in ZK
> and on subsequent restarts, the consumer starts consumption from the offset
> stored in ZK.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 29, 2012 at 9:54 AM, Elben Shira <elbenshira@gmail.com> wrote:
>
>> Hi Neha,
>>
>> What does it mean to fetch the latest data? If there are items in the kafka
>> queue and no new items are coming in, would autooffset.reset="largest" wait
>> for new items or would it consume some of the existing items in the queue?
>>
>> I'm asking this because I've modified the sensei gateways
>> to autooffset.reset="largest" but it's still consuming items previously in
>> the queue, and I'm not sure if the problem is on my end.
>>
>> I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find the
>> latest offset for the queue itself (not the consumer)?
>>
>> Elben
>>
>>
>> On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkhede@gmail.com
>> >wrote:
>>
>> > John,
>> >
>> > You can achieve that via the autooffset.reset="largest" config option
>> > in the consumer.
>> >
>> > Thanks,
>> > Neha
>> >
>> > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <john.wang@gmail.com> wrote:
>> > > Hi Neha:
>> > >
>> > >     Can you point me to the api to always get data from tail?
>> > >
>> > > Thanks
>> > >
>> > > -John
>> > >
>> > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <
>> neha.narkhede@gmail.com
>> > >wrote:
>> > >
>> > >> Elben,
>> > >>
>> > >> Or sensei can provide a restart from tail API, that will delete the
>> > offset
>> > >> state from ZK and restart its Kafka consumer. Currently that is the
>> best
>> > >> way to have a Kafka consumer restart fetching latest data.
>> > >>
>> > >> Thanks,
>> > >> Neha
>> > >> On Mar 22, 2012 4:39 PM, "John Wang" <john.wang@gmail.com> wrote:
>> > >>
>> > >> > I think in the newer kafka release, there is a tool to update
the
>> > version
>> > >> > for a client.
>> > >> >
>> > >> > Jun, can you elaborate on that?
>> > >> >
>> > >> > Thanks
>> > >> >
>> > >> > -John
>> > >> >
>> > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <elbenshira@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > > We're using use kafka+sensei at our company and we'd love
to be
>> > able to
>> > >> > > start at the end of the queue (latest entries). The use case
is
>> when
>> > >> > sensei
>> > >> > > goes down or has to be restarted, we only care about indexing
the
>> > >> latest
>> > >> > > items.
>> > >> > >
>> > >> > > A workaround I guess is to stop sensei, manually change the
high
>> > >> > watermark
>> > >> > > in kafka, then restart sensei.
>> > >> > >
>> > >> > > Elben
>> > >> > >
>> > >> > >
>> > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <junrao@gmail.com>
>> wrote:
>> > >> > >
>> > >> > > > Casey,
>> > >> > > >
>> > >> > > > You can't get offsets using zk-based consumer now. We
are still
>> > >> trying
>> > >> > to
>> > >> > > > figure about how to support that in the future.
>> > >> > > >
>> > >> > > > Thanks,
>> > >> > > >
>> > >> > > > Jun
>> > >> > > >
>> > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
>> > >> > > > Casey.Sybrandy@six3systems.com> wrote:
>> > >> > > >
>> > >> > > > > Hello,
>> > >> > > > >
>> > >> > > > > Just out of curiosity, how would you get the offsets
when
>> using
>> > a
>> > >> > > > zk-based
>> > >> > > > > consumer?  I don't see anything that seems to
let you do this
>> > >> easily.
>> > >> > > > >
>> > >> > > > > Casey
>> > >> > > > >
>> > >> > > > > ________________________________________
>> > >> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
>> > >> > > > > Sent: Monday, March 12, 2012 5:51 PM
>> > >> > > > > To: kafka-users@incubator.apache.org
>> > >> > > > > Subject: Re: Replay
>> > >> > > > >
>> > >> > > > > With simple consumer, you can start from any valid
offset that
>> > you
>> > >> > keep
>> > >> > > > > track off.
>> > >> > > > >
>> > >> > > > > With the high level (zk-based) consumer: you will
need to stop
>> > all
>> > >> > > > > consumers, and then update the offsets in zookeeper
(see the
>> > >> > > > > ImportZkOffsets tool in the tools package) with
the offsets
>> you
>> > >> want
>> > >> > to
>> > >> > > > > replay from before restarting your consumer.
>> > >> > > > >
>> > >> > > > > Thanks,
>> > >> > > > >
>> > >> > > > > Joel
>> > >> > > > >
>> > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <
>> > JainK@bit-sys.com>
>> > >> > > wrote:
>> > >> > > > >
>> > >> > > > > > Hi,
>> > >> > > > > >
>> > >> > > > > > Sorry, new to Kafka. Is it possible for a
consumer to replay
>> > >> > messages
>> > >> > > > > from
>> > >> > > > > > an "offset" that may have been committed before?
>> > >> > > > > >
>> > >> > > > > > Thanks for the help,
>> > >> > > > > > Kokil Jain
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>

Mime
View raw message