kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Replay
Date Fri, 23 Mar 2012 02:05:27 GMT
John,

You can achieve that via the autooffset.reset="largest" config option
in the consumer.

Thanks,
Neha

On Thu, Mar 22, 2012 at 6:15 PM, John Wang <john.wang@gmail.com> wrote:
> Hi Neha:
>
>     Can you point me to the api to always get data from tail?
>
> Thanks
>
> -John
>
> On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:
>
>> Elben,
>>
>> Or sensei can provide a restart from tail API, that will delete the offset
>> state from ZK and restart its Kafka consumer. Currently that is the best
>> way to have a Kafka consumer restart fetching latest data.
>>
>> Thanks,
>> Neha
>> On Mar 22, 2012 4:39 PM, "John Wang" <john.wang@gmail.com> wrote:
>>
>> > I think in the newer kafka release, there is a tool to update the version
>> > for a client.
>> >
>> > Jun, can you elaborate on that?
>> >
>> > Thanks
>> >
>> > -John
>> >
>> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <elbenshira@gmail.com>
>> wrote:
>> >
>> > > We're using use kafka+sensei at our company and we'd love to be able to
>> > > start at the end of the queue (latest entries). The use case is when
>> > sensei
>> > > goes down or has to be restarted, we only care about indexing the
>> latest
>> > > items.
>> > >
>> > > A workaround I guess is to stop sensei, manually change the high
>> > watermark
>> > > in kafka, then restart sensei.
>> > >
>> > > Elben
>> > >
>> > >
>> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <junrao@gmail.com> wrote:
>> > >
>> > > > Casey,
>> > > >
>> > > > You can't get offsets using zk-based consumer now. We are still
>> trying
>> > to
>> > > > figure about how to support that in the future.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jun
>> > > >
>> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey <
>> > > > Casey.Sybrandy@six3systems.com> wrote:
>> > > >
>> > > > > Hello,
>> > > > >
>> > > > > Just out of curiosity, how would you get the offsets when using
a
>> > > > zk-based
>> > > > > consumer?  I don't see anything that seems to let you do this
>> easily.
>> > > > >
>> > > > > Casey
>> > > > >
>> > > > > ________________________________________
>> > > > > From: Joel Koshy [jjkoshy.w@gmail.com]
>> > > > > Sent: Monday, March 12, 2012 5:51 PM
>> > > > > To: kafka-users@incubator.apache.org
>> > > > > Subject: Re: Replay
>> > > > >
>> > > > > With simple consumer, you can start from any valid offset that
you
>> > keep
>> > > > > track off.
>> > > > >
>> > > > > With the high level (zk-based) consumer: you will need to stop
all
>> > > > > consumers, and then update the offsets in zookeeper (see the
>> > > > > ImportZkOffsets tool in the tools package) with the offsets you
>> want
>> > to
>> > > > > replay from before restarting your consumer.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Joel
>> > > > >
>> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil <JainK@bit-sys.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay
>> > messages
>> > > > > from
>> > > > > > an "offset" that may have been committed before?
>> > > > > >
>> > > > > > Thanks for the help,
>> > > > > > Kokil Jain
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Mime
View raw message