kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Becket Qin <becket....@gmail.com>
Subject Re: Checking the consumer lag when using manual partition assignment with the KafkaConsumer
Date Mon, 07 Nov 2016 16:52:43 GMT
Mattias,

> kafka-consumer-offet-checker works for older Kakfa versions for which
> offsets are stored in ZK. It does not work for v0.9+ when offsets are
> stored in topic __consumer_offsets

That is not true. The offset checker works with both ZK based and broker
based offset commit. It query broker first to see if there is committed
offset for the partition, if there isn't, it then fall back to looking at
ZK.

Thanks,

Jiangjie (Becket) Qin

On Sun, Nov 6, 2016 at 6:36 PM, Matthias J. Sax <matthias@confluent.io>
wrote:

> kafka-consumer-offet-checker works for older Kakfa versions for which
> offsets are stored in ZK. It does not work for v0.9+ when offsets are
> stored in topic __consumer_offsets
>
> Robert, you are right that kafka-consumer-groups only show information
> of active consumer groups...
>
> @Vincent, I am not sure if this is a bug or not. If yes, we should make
> sure that there is a ticket... I'll double check.
>
> -Matthias
>
> On 11/06/2016 05:39 PM, Becket Qin wrote:
> > Hi Robert,
> >
> > Have you tried Kafka-consumer-offset-checker.sh come with Kafka? It
> should
> > work regardless whether Kafka based group management is used or not.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Sun, Nov 6, 2016 at 4:00 PM, Vincent Dautremont <
> > vincent.dautremont@olamobile.com> wrote:
> >
> >> By the way I remember having read somewhere on this list that this
> utility
> >> not showing info for consumer groups that do not have current active
> >> consumers was a bug .
> >>
> >> That would be a thing to fix, is there an expected fix date / fix
> release
> >> for this?
> >>
> >>
> >>> Le 6 nov. 2016 à 13:23, Robert Metzger <rmetzger@apache.org> a écrit
:
> >>>
> >>> Hi Matthias,
> >>>
> >>> the bin/kafka-consumer-groups.sh utility is exactly what the user in
> >>> http://grokbase.com/t/kafka/users/163rrq9ne8/new-consumer-
> >> group-not-showing-up
> >>> has used.
> >>> It seems that the broker is not returning consumer groups that don't
> >>> participate in the consumer group balancing mechanism.
> >>>
> >>>
> >>> On Fri, Nov 4, 2016 at 6:07 PM, Matthias J. Sax <matthias@confluent.io
> >
> >>> wrote:
> >>>
> > Robert,
> >
> > you can use bin/kafka-consumer-groups.sh instead.
> >
> >>>>>> bin/kafka-consumer-offset-checker.sh [2016-11-04 10:05:07,852]
WARN
> >>>>>> WARNING: ConsumerOffsetChecker is deprecated and will be dropped
in
> >>>>>> releases following 0.9.0. Use ConsumerGroupCommand instead.
> >>>>>> (kafka.tools.ConsumerOffsetChecker$)
> >
> >
> > -Matthias
> >
> >>>>>> On 11/3/16 4:07 AM, Robert Metzger wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> some Flink users recently noticed that they can not check the
> >>>>>> consumer lag when using Flink's kafka consumer [1]. According
to
> >>>>>> this discussion on the Kafka user list [2] the
> >>>>>> kafka-consumer-groups.sh utility doesn't work with KafkaConsumers
> >>>>>> with manual partition assignment.
> >>>>>>
> >>>>>> Is there a way to get the consumer lag for a consumer that
> >>>>>> regularly commits its offsets to the broker?
> >>>>>>
> >>>>>> Regards, Robert
> >>>>>>
> >>>>>>
> >>>>>> [1] https://issues.apache.org/jira/browse/FLINK-5001 [2]
> >>>>>> http://grokbase.com/t/kafka/users/163rrq9ne8/new-consumer-
> > group-not-showing-up
> >>>>>>
> >>>>
> >>
> >> --
> >> The information transmitted is intended only for the person or entity to
> >> which it is addressed and may contain confidential and/or privileged
> >> material. Any review, retransmission, dissemination or other use of, or
> >> taking of any action in reliance upon, this information by persons or
> >> entities other than the intended recipient is prohibited. If you
> received
> >> this in error, please contact the sender and delete the material from
> any
> >> computer.
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message