kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: How to reset a consumer-group's offset in kafka 0.9?
Date Mon, 04 Jan 2016 10:12:44 GMT
Created related ticket https://issues.apache.org/jira/browse/KAFKA-3057
Maybe request for extra operations support/tools could be a separate ticket.

On Mon, Jan 4, 2016 at 10:48 AM, Han JU <ju.han.felix@gmail.com> wrote:

> Thanks a lot Guozhang!
> So currently there's no way to delete a consumer group in with the new
> consumer API? Do you plan to add it in the next versions?
>
>
> 2015-12-31 20:09 GMT+01:00 Guozhang Wang <wangguoz@gmail.com>:
>
> > Hello Han,
> >
> > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to
> reset
> > your consuming offsets. Or if you are stopping the consumer between your
> > test runs you can also commit() with offset 0 before you closing your
> > consumer at the end of each test.
> >
> > 2. In the 0.9 Java consumer both the consumer registry information and
> the
> > offsets are stored in Kafka servers instead of on the ZK, you can find a
> > more detailed description and motivation of this change here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> >
> > So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing the
> > consumer registry information.
> >
> > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see
> > "deprecations in 0.9.0" here:
> >
> > http://kafka.apache.org/documentation.html#upgrade_9_breaking
> >
> > Instead you can try to use bin/kafka-consumer-groups.sh
> > (kafka.admin.ConsumerGroupCommand).
> >
> > Guozhang
> >
> >
> >
> > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju.han.felix@gmail.com> wrote:
> >
> > > Hi Marko,
> > >
> > > Yes we're currently using this on our production kafka 0.8. But it does
> > not
> > > seem to work with the new consumer API in 0.9.
> > > To answer my own question about deleting consumer group in new consumer
> > > API, it seems that it's currently not possible with the new consumer
> API
> > > (there's no delete related method in `AdminClient` of the new consumer
> > > API).
> > >
> > >
> > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <marko.bonaci@sematext.com>:
> > >
> > > > If you want to monitor offset (ZK or Kafka based), try with
> QuantFind's
> > > > Kafka Offset Monitor.
> > > > If you use Docker, it's easy as:
> > > >
> > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > > > jpodeszwik/kafka-offset-monitor
> > > > and then opening browser to dockerhost:8080.
> > > >
> > > > If not in the Docker mood, use instructions here:
> > > > https://github.com/quantifind/KafkaOffsetMonitor
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju.han.felix@gmail.com>
> > wrote:
> > > >
> > > > > Thanks guys. The `seek` seems a solution. But it's more cumbersome
> > than
> > > > in
> > > > > 0.8 because I have to plug in some extra code in my consumer
> > > abstractions
> > > > > rather than simply deleting a zk node.
> > > > > And one more question: where does kafka 0.9 stores the
> consumer-group
> > > > > information? In fact I also tried to delete the consumer group but
> > the
> > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9.
> > And
> > > > also
> > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> --zookeeper
> > > > > localhost:2181 --group group-name` seems broken.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <marko.bonaci@sematext.com
> >:
> > > > >
> > > > > > I was refering to Dana Powers's answer in the link I posted
(to
> > use a
> > > > > > client API). You can find an example here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > > > >
> > > > > > Marko Bonaći
> > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support
> > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > <http://sematext.com/about/contact.html>
> > > > > >
> > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <sslavic@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Then I guess @Before test, explicitly commit offset of
0.
> > > > > > >
> > > > > > > There doesn't seem to be a tool for committing offset,
only for
> > > > > > > checking/fetching current offset (see
> > > > > > > http://kafka.apache.org/documentation.html#operations )
> > > > > > >
> > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <
> ju.han.felix@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Stevo,
> > > > > > > >
> > > > > > > > But by deleting and recreating the topic, do I remove
also
> the
> > > > > messages
> > > > > > > > ingested?
> > > > > > > > My use case is that I ingest prepared messages once
and run
> > > > consumer
> > > > > > > tests
> > > > > > > > multiple times, between each test run I reset the
consumer
> > > group's
> > > > > > offset
> > > > > > > > so that each run starts from the beginning and consumers
all
> > the
> > > > > > > messages.
> > > > > > > >
> > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <sslavic@gmail.com>:
> > > > > > > >
> > > > > > > > > Have you considered deleting and recreating topic
used in
> > test?
> > > > > > > > > Once topic is clean, read/poll once - any committed
offset
> > > should
> > > > > be
> > > > > > > > > outside of the range, and consumer should reset
offset.
> > > > > > > > >
> > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> > > ju.han.felix@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > > For local test purpose I need to frequently
reset offset
> > for
> > > a
> > > > > > > consumer
> > > > > > > > > > group. In 0.8 I just delete the consumer
group's zk node
> > > under
> > > > > > > > > > `/consumers`. But with the redesign of the
0.9, how
> could I
> > > > > achieve
> > > > > > > the
> > > > > > > > > > same thing?
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *JU Han*
> > > > > > > > > >
> > > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > > >
> > > > > > > > > > +33 0619608888
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *JU Han*
> > > > > > > >
> > > > > > > > Software Engineer @ Teads.tv
> > > > > > > >
> > > > > > > > +33 0619608888
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *JU Han*
> > > > >
> > > > > Software Engineer @ Teads.tv
> > > > >
> > > > > +33 0619608888
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message