kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Bonaći <marko.bon...@sematext.com>
Subject Re: how to reset kafka offset in zookeeper
Date Fri, 18 Dec 2015 17:38:42 GMT
Hmm, I guess you're right Tod :)
Just to confirm, you meant that, while you're changing the exported file it
might happen that one of the segment files becomes eligible for cleanup by
retention, which would then make the imported offsets out of range?

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tpalino@gmail.com> wrote:

> That works if you want to set to an arbitrary offset, Marko. However in the
> case the OP described, wanting to reset to smallest, it is better to just
> delete the consumer group and start the consumer with auto.offset.reset set
> to smallest. The reason is that while you can pull the current smallest
> offsets from the brokers and set them in Zookeeper for the consumer, by the
> time you do that the smallest offset is likely no longer valid. This means
> you’re going to resort to the offset reset logic anyways.
>
> -Todd
>
>
> On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <marko.bonaci@sematext.com>
> wrote:
>
> > You can also do this:
> > 1. stop consumers
> > 2. export offsets from ZK
> > 3. make changes to the exported file
> > 4. import offsets to ZK
> > 5. start consumers
> >
> > e.g.
> > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group group-name
> > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > /tmp/zk-offsets --zkconnect localhost:2181
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <jens.rantil@tink.se>
> wrote:
> >
> > > Hi,
> > >
> > > I noticed that a consumer in the new consumer API supports setting the
> > > offset for a partition to beginning. I assume doing so also would
> update
> > > the offset in Zookeeper eventually.
> > >
> > > Cheers,
> > > Jens
> > >
> > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > pathodia.akhilesh@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I want to reset the kafka offset in zookeeper so that the consumer
> will
> > > > start reading messages from first offset. I am using flume as a
> > consumer
> > > to
> > > > kafka. I have set the kafka property kafka.auto.offset.reset to
> > > "smallest",
> > > > but it does not reset the offset in zookeeper and that's why flume
> will
> > > not
> > > > read messages from first offset.
> > > >
> > > > Is there any way to reset kafka offset in zookeeper?
> > > >
> > > > Thanks,
> > > > Akhilesh
> > > >
> > >
> > >
> > > --
> > > Jens Rantil
> > > Backend engineer
> > > Tink AB
> > >
> > > Email: jens.rantil@tink.se
> > > Phone: +46 708 84 18 32
> > > Web: www.tink.se
> > >
> > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > <
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > >
> > >  Twitter <https://twitter.com/tink>
> > >
> >
>
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message