kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhilesh Pathodia <pathodia.akhil...@gmail.com>
Subject Re: how to reset kafka offset in zookeeper
Date Sat, 19 Dec 2015 19:34:48 GMT
What is the command  to delete  group from zookeeper? I dont find
/consumer/ directory? I am using cloudera, is there any place on cloudera
manager where I can delete the group?

Thanks

On Sat, Dec 19, 2015 at 11:47 PM, Todd Palino <tpalino@gmail.com> wrote:

> If what you want to do is reset to smallest, all you need to do is stop the
> consumer, delete the group from Zookeeper, and restart the consumer. It
> will automatically create the group again.
>
> You only need to export the offsets first if you later need to reset to
> where you were in the partitions.
>
> -Todd
>
> On Saturday, December 19, 2015, Akhilesh Pathodia <
> pathodia.akhilesh@gmail.com> wrote:
>
> > What is the process for deleting the consumer group from zookeeper?
> Should
> > I export offset, delete and then import?
> >
> > Thanks,
> > Akhilesh
> >
> > On Fri, Dec 18, 2015 at 11:32 PM, Todd Palino <tpalino@gmail.com
> > <javascript:;>> wrote:
> >
> > > Yes, that’s right. It’s just work for no real gain :)
> > >
> > > -Todd
> > >
> > > On Fri, Dec 18, 2015 at 9:38 AM, Marko Bonaći <
> marko.bonaci@sematext.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > Hmm, I guess you're right Tod :)
> > > > Just to confirm, you meant that, while you're changing the exported
> > file
> > > it
> > > > might happen that one of the segment files becomes eligible for
> cleanup
> > > by
> > > > retention, which would then make the imported offsets out of range?
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tpalino@gmail.com
> > <javascript:;>> wrote:
> > > >
> > > > > That works if you want to set to an arbitrary offset, Marko.
> However
> > in
> > > > the
> > > > > case the OP described, wanting to reset to smallest, it is better
> to
> > > just
> > > > > delete the consumer group and start the consumer with
> > auto.offset.reset
> > > > set
> > > > > to smallest. The reason is that while you can pull the current
> > smallest
> > > > > offsets from the brokers and set them in Zookeeper for the
> consumer,
> > by
> > > > the
> > > > > time you do that the smallest offset is likely no longer valid.
> This
> > > > means
> > > > > you’re going to resort to the offset reset logic anyways.
> > > > >
> > > > > -Todd
> > > > >
> > > > >
> > > > > On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <
> > > marko.bonaci@sematext.com <javascript:;>
> > > > >
> > > > > wrote:
> > > > >
> > > > > > You can also do this:
> > > > > > 1. stop consumers
> > > > > > 2. export offsets from ZK
> > > > > > 3. make changes to the exported file
> > > > > > 4. import offsets to ZK
> > > > > > 5. start consumers
> > > > > >
> > > > > > e.g.
> > > > > > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group
> > group-name
> > > > > > --output-file /tmp/zk-offsets --zkconnect localhost:2181
> > > > > > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file
> > > > > > /tmp/zk-offsets --zkconnect localhost:2181
> > > > > >
> > > > > > Marko Bonaći
> > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support
> > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > <http://sematext.com/about/contact.html>
> > > > > >
> > > > > > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <
> jens.rantil@tink.se
> > <javascript:;>>
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I noticed that a consumer in the new consumer API supports
> > setting
> > > > the
> > > > > > > offset for a partition to beginning. I assume doing so
also
> would
> > > > > update
> > > > > > > the offset in Zookeeper eventually.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jens
> > > > > > >
> > > > > > > On Friday, December 18, 2015, Akhilesh Pathodia <
> > > > > > > pathodia.akhilesh@gmail.com <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I want to reset the kafka offset in zookeeper so that
the
> > > consumer
> > > > > will
> > > > > > > > start reading messages from first offset. I am using
flume
> as a
> > > > > > consumer
> > > > > > > to
> > > > > > > > kafka. I have set the kafka property kafka.auto.offset.reset
> to
> > > > > > > "smallest",
> > > > > > > > but it does not reset the offset in zookeeper and
that's why
> > > flume
> > > > > will
> > > > > > > not
> > > > > > > > read messages from first offset.
> > > > > > > >
> > > > > > > > Is there any way to reset kafka offset in zookeeper?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Akhilesh
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Jens Rantil
> > > > > > > Backend engineer
> > > > > > > Tink AB
> > > > > > >
> > > > > > > Email: jens.rantil@tink.se <javascript:;>
> > > > > > > Phone: +46 708 84 18 32
> > > > > > > Web: www.tink.se
> > > > > > >
> > > > > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
> > > > > > > >
> > > > > > >  Twitter <https://twitter.com/tink>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *—-*
> > > > > *Todd Palino*
> > > > > Staff Site Reliability Engineer
> > > > > Data Infrastructure Streaming
> > > > >
> > > > >
> > > > >
> > > > > linkedin.com/in/toddpalino
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *—-*
> > > *Todd Palino*
> > > Staff Site Reliability Engineer
> > > Data Infrastructure Streaming
> > >
> > >
> > >
> > > linkedin.com/in/toddpalino
> > >
> >
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message