kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Bonaći <marko.bon...@sematext.com>
Subject Re: How to reset a consumer-group's offset in kafka 0.9?
Date Wed, 30 Dec 2015 18:40:51 GMT
Hi Han,
if it doesn't work you should file an issue, since it explicitly says in
the readme that it works with:
  1 zookeeper built-in high-level consumer (based on Zookeeper)
  2 kafka built-in offset management API (based on Kafka internal topic)
  3 Storm Kafka Spout (based on Zookeeper by default)

I'm still on Kafka 0.8, so I can't shed any light on your issue.
Thx for the AdminClient info.

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Wed, Dec 30, 2015 at 6:10 PM, Han JU <ju.han.felix@gmail.com> wrote:

> Hi Marko,
>
> Yes we're currently using this on our production kafka 0.8. But it does not
> seem to work with the new consumer API in 0.9.
> To answer my own question about deleting consumer group in new consumer
> API, it seems that it's currently not possible with the new consumer API
> (there's no delete related method in `AdminClient` of the new consumer
> API).
>
>
> 2015-12-30 17:02 GMT+01:00 Marko Bonaći <marko.bonaci@sematext.com>:
>
> > If you want to monitor offset (ZK or Kafka based), try with QuantFind's
> > Kafka Offset Monitor.
> > If you use Docker, it's easy as:
> >
> > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > jpodeszwik/kafka-offset-monitor
> > and then opening browser to dockerhost:8080.
> >
> > If not in the Docker mood, use instructions here:
> > https://github.com/quantifind/KafkaOffsetMonitor
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju.han.felix@gmail.com> wrote:
> >
> > > Thanks guys. The `seek` seems a solution. But it's more cumbersome than
> > in
> > > 0.8 because I have to plug in some extra code in my consumer
> abstractions
> > > rather than simply deleting a zk node.
> > > And one more question: where does kafka 0.9 stores the consumer-group
> > > information? In fact I also tried to delete the consumer group but the
> > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. And
> > also
> > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> > > localhost:2181 --group group-name` seems broken.
> > >
> > > Thanks!
> > >
> > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <marko.bonaci@sematext.com>:
> > >
> > > > I was refering to Dana Powers's answer in the link I posted (to use a
> > > > client API). You can find an example here:
> > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <sslavic@gmail.com>
> > wrote:
> > > >
> > > > > Then I guess @Before test, explicitly commit offset of 0.
> > > > >
> > > > > There doesn't seem to be a tool for committing offset, only for
> > > > > checking/fetching current offset (see
> > > > > http://kafka.apache.org/documentation.html#operations )
> > > > >
> > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju.han.felix@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Stevo,
> > > > > >
> > > > > > But by deleting and recreating the topic, do I remove also the
> > > messages
> > > > > > ingested?
> > > > > > My use case is that I ingest prepared messages once and run
> > consumer
> > > > > tests
> > > > > > multiple times, between each test run I reset the consumer
> group's
> > > > offset
> > > > > > so that each run starts from the beginning and consumers all
the
> > > > > messages.
> > > > > >
> > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <sslavic@gmail.com>:
> > > > > >
> > > > > > > Have you considered deleting and recreating topic used
in test?
> > > > > > > Once topic is clean, read/poll once - any committed offset
> should
> > > be
> > > > > > > outside of the range, and consumer should reset offset.
> > > > > > >
> > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> ju.han.felix@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > For local test purpose I need to frequently reset
offset for
> a
> > > > > consumer
> > > > > > > > group. In 0.8 I just delete the consumer group's zk
node
> under
> > > > > > > > `/consumers`. But with the redesign of the 0.9, how
could I
> > > achieve
> > > > > the
> > > > > > > > same thing?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > --
> > > > > > > > *JU Han*
> > > > > > > >
> > > > > > > > Software Engineer @ Teads.tv
> > > > > > > >
> > > > > > > > +33 0619608888
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *JU Han*
> > > > > >
> > > > > > Software Engineer @ Teads.tv
> > > > > >
> > > > > > +33 0619608888
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
>
>
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message