kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: How to reset a consumer-group's offset in kafka 0.9?
Date Mon, 04 Jan 2016 17:00:56 GMT
What you really want is to remove the committed offsets in Kafka servers,
not the consumer group registry metadata (e.g. such as partition assignment
information, etc), which will not be done immediately after the group is
removed, but only after the offsets expiration has elapsed.

We can add this feature as an admin command in future versions.

Guozhang


On Mon, Jan 4, 2016 at 2:12 AM, Stevo Slavić <sslavic@gmail.com> wrote:

> Created related ticket https://issues.apache.org/jira/browse/KAFKA-3057
> Maybe request for extra operations support/tools could be a separate
> ticket.
>
> On Mon, Jan 4, 2016 at 10:48 AM, Han JU <ju.han.felix@gmail.com> wrote:
>
> > Thanks a lot Guozhang!
> > So currently there's no way to delete a consumer group in with the new
> > consumer API? Do you plan to add it in the next versions?
> >
> >
> > 2015-12-31 20:09 GMT+01:00 Guozhang Wang <wangguoz@gmail.com>:
> >
> > > Hello Han,
> > >
> > > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to
> > reset
> > > your consuming offsets. Or if you are stopping the consumer between
> your
> > > test runs you can also commit() with offset 0 before you closing your
> > > consumer at the end of each test.
> > >
> > > 2. In the 0.9 Java consumer both the consumer registry information and
> > the
> > > offsets are stored in Kafka servers instead of on the ZK, you can find
> a
> > > more detailed description and motivation of this change here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > >
> > > So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing
> the
> > > consumer registry information.
> > >
> > > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see
> > > "deprecations in 0.9.0" here:
> > >
> > > http://kafka.apache.org/documentation.html#upgrade_9_breaking
> > >
> > > Instead you can try to use bin/kafka-consumer-groups.sh
> > > (kafka.admin.ConsumerGroupCommand).
> > >
> > > Guozhang
> > >
> > >
> > >
> > > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju.han.felix@gmail.com>
> wrote:
> > >
> > > > Hi Marko,
> > > >
> > > > Yes we're currently using this on our production kafka 0.8. But it
> does
> > > not
> > > > seem to work with the new consumer API in 0.9.
> > > > To answer my own question about deleting consumer group in new
> consumer
> > > > API, it seems that it's currently not possible with the new consumer
> > API
> > > > (there's no delete related method in `AdminClient` of the new
> consumer
> > > > API).
> > > >
> > > >
> > > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <marko.bonaci@sematext.com>:
> > > >
> > > > > If you want to monitor offset (ZK or Kafka based), try with
> > QuantFind's
> > > > > Kafka Offset Monitor.
> > > > > If you use Docker, it's easy as:
> > > > >
> > > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > > > > jpodeszwik/kafka-offset-monitor
> > > > > and then opening browser to dockerhost:8080.
> > > > >
> > > > > If not in the Docker mood, use instructions here:
> > > > > https://github.com/quantifind/KafkaOffsetMonitor
> > > > >
> > > > > Marko Bonaći
> > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support
> > > > > Sematext <http://sematext.com/> | Contact
> > > > > <http://sematext.com/about/contact.html>
> > > > >
> > > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju.han.felix@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thanks guys. The `seek` seems a solution. But it's more
> cumbersome
> > > than
> > > > > in
> > > > > > 0.8 because I have to plug in some extra code in my consumer
> > > > abstractions
> > > > > > rather than simply deleting a zk node.
> > > > > > And one more question: where does kafka 0.9 stores the
> > consumer-group
> > > > > > information? In fact I also tried to delete the consumer group
> but
> > > the
> > > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in
> 0.9.
> > > And
> > > > > also
> > > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> > --zookeeper
> > > > > > localhost:2181 --group group-name` seems broken.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <
> marko.bonaci@sematext.com
> > >:
> > > > > >
> > > > > > > I was refering to Dana Powers's answer in the link I posted
(to
> > > use a
> > > > > > > client API). You can find an example here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > > > > >
> > > > > > > Marko Bonaći
> > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized
Log
> > > > Management
> > > > > > > Solr & Elasticsearch Support
> > > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > > <http://sematext.com/about/contact.html>
> > > > > > >
> > > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <
> sslavic@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Then I guess @Before test, explicitly commit offset
of 0.
> > > > > > > >
> > > > > > > > There doesn't seem to be a tool for committing offset,
only
> for
> > > > > > > > checking/fetching current offset (see
> > > > > > > > http://kafka.apache.org/documentation.html#operations
)
> > > > > > > >
> > > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <
> > ju.han.felix@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Stevo,
> > > > > > > > >
> > > > > > > > > But by deleting and recreating the topic, do
I remove also
> > the
> > > > > > messages
> > > > > > > > > ingested?
> > > > > > > > > My use case is that I ingest prepared messages
once and run
> > > > > consumer
> > > > > > > > tests
> > > > > > > > > multiple times, between each test run I reset
the consumer
> > > > group's
> > > > > > > offset
> > > > > > > > > so that each run starts from the beginning and
consumers
> all
> > > the
> > > > > > > > messages.
> > > > > > > > >
> > > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <sslavic@gmail.com
> >:
> > > > > > > > >
> > > > > > > > > > Have you considered deleting and recreating
topic used in
> > > test?
> > > > > > > > > > Once topic is clean, read/poll once - any
committed
> offset
> > > > should
> > > > > > be
> > > > > > > > > > outside of the range, and consumer should
reset offset.
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU
<
> > > > ju.han.felix@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello,
> > > > > > > > > > >
> > > > > > > > > > > For local test purpose I need to frequently
reset
> offset
> > > for
> > > > a
> > > > > > > > consumer
> > > > > > > > > > > group. In 0.8 I just delete the consumer
group's zk
> node
> > > > under
> > > > > > > > > > > `/consumers`. But with the redesign
of the 0.9, how
> > could I
> > > > > > achieve
> > > > > > > > the
> > > > > > > > > > > same thing?
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *JU Han*
> > > > > > > > > > >
> > > > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > > > >
> > > > > > > > > > > +33 0619608888
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *JU Han*
> > > > > > > > >
> > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > >
> > > > > > > > > +33 0619608888
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *JU Han*
> > > > > >
> > > > > > Software Engineer @ Teads.tv
> > > > > >
> > > > > > +33 0619608888
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *JU Han*
> > > >
> > > > Software Engineer @ Teads.tv
> > > >
> > > > +33 0619608888
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > *JU Han*
> >
> > Software Engineer @ Teads.tv
> >
> > +33 0619608888
> >
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message