kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Dautremont <vincent.dautrem...@olamobile.com>
Subject Re: Checking the consumer lag when using manual partition assignment with the KafkaConsumer
Date Tue, 15 Nov 2016 15:49:15 GMT
Hi Matthias,
from what I understand it is (was) a bug
https://issues.apache.org/jira/browse/KAFKA-3859?focusedCommentId=15376317&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15376317

But I see it would have been fixed late october
https://issues.apache.org/jira/browse/KAFKA-3853
as I'm using 10.0.0 I haven't yet check if it is fixed in the last release.

Vincent.

On Mon, Nov 7, 2016 at 3:36 AM, Matthias J. Sax <matthias@confluent.io>
wrote:

> kafka-consumer-offet-checker works for older Kakfa versions for which
> offsets are stored in ZK. It does not work for v0.9+ when offsets are
> stored in topic __consumer_offsets
>
> Robert, you are right that kafka-consumer-groups only show information
> of active consumer groups...
>
> @Vincent, I am not sure if this is a bug or not. If yes, we should make
> sure that there is a ticket... I'll double check.
>
> -Matthias
>
> On 11/06/2016 05:39 PM, Becket Qin wrote:
> > Hi Robert,
> >
> > Have you tried Kafka-consumer-offset-checker.sh come with Kafka? It
> should
> > work regardless whether Kafka based group management is used or not.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Sun, Nov 6, 2016 at 4:00 PM, Vincent Dautremont <
> > vincent.dautremont@olamobile.com> wrote:
> >
> >> By the way I remember having read somewhere on this list that this
> utility
> >> not showing info for consumer groups that do not have current active
> >> consumers was a bug .
> >>
> >> That would be a thing to fix, is there an expected fix date / fix
> release
> >> for this?
> >>
> >>
> >>> Le 6 nov. 2016 à 13:23, Robert Metzger <rmetzger@apache.org> a écrit
:
> >>>
> >>> Hi Matthias,
> >>>
> >>> the bin/kafka-consumer-groups.sh utility is exactly what the user in
> >>> http://grokbase.com/t/kafka/users/163rrq9ne8/new-consumer-
> >> group-not-showing-up
> >>> has used.
> >>> It seems that the broker is not returning consumer groups that don't
> >>> participate in the consumer group balancing mechanism.
> >>>
> >>>
> >>> On Fri, Nov 4, 2016 at 6:07 PM, Matthias J. Sax <matthias@confluent.io
> >
> >>> wrote:
> >>>
> > Robert,
> >
> > you can use bin/kafka-consumer-groups.sh instead.
> >
> >>>>>> bin/kafka-consumer-offset-checker.sh [2016-11-04 10:05:07,852]
WARN
> >>>>>> WARNING: ConsumerOffsetChecker is deprecated and will be dropped
in
> >>>>>> releases following 0.9.0. Use ConsumerGroupCommand instead.
> >>>>>> (kafka.tools.ConsumerOffsetChecker$)
> >
> >
> > -Matthias
> >
> >>>>>> On 11/3/16 4:07 AM, Robert Metzger wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> some Flink users recently noticed that they can not check the
> >>>>>> consumer lag when using Flink's kafka consumer [1]. According
to
> >>>>>> this discussion on the Kafka user list [2] the
> >>>>>> kafka-consumer-groups.sh utility doesn't work with KafkaConsumers
> >>>>>> with manual partition assignment.
> >>>>>>
> >>>>>> Is there a way to get the consumer lag for a consumer that
> >>>>>> regularly commits its offsets to the broker?
> >>>>>>
> >>>>>> Regards, Robert
> >>>>>>
> >>>>>>
> >>>>>> [1] https://issues.apache.org/jira/browse/FLINK-5001 [2]
> >>>>>> http://grokbase.com/t/kafka/users/163rrq9ne8/new-consumer-
> > group-not-showing-up
> >>>>>>
> >>>>
> >>
> >> --
> >> The information transmitted is intended only for the person or entity to
> >> which it is addressed and may contain confidential and/or privileged
> >> material. Any review, retransmission, dissemination or other use of, or
> >> taking of any action in reliance upon, this information by persons or
> >> entities other than the intended recipient is prohibited. If you
> received
> >> this in error, please contact the sender and delete the material from
> any
> >> computer.
> >>
> >
>
>

-- 
The information transmitted is intended only for the person or entity to 
which it is addressed and may contain confidential and/or privileged 
material. Any review, retransmission, dissemination or other use of, or 
taking of any action in reliance upon, this information by persons or 
entities other than the intended recipient is prohibited. If you received 
this in error, please contact the sender and delete the material from any 
computer.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message