kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: checking consumer lag on KStreams app?
Date Tue, 13 Dec 2016 17:36:13 GMT
There is a workaround:

http://stackoverflow.com/questions/41097126/how-to-get-the-group-commit-offset-from-kafka0-10-x

I never had time to follow up with this issue. Will put it back on the
agenda. Should really get fixed.


-Matthias

On 12/13/16 2:03 AM, Sachin Mittal wrote:
> If this is a bug then it is not fixed because I just build kafka from
> source and it gave me the reported error.
> 
> On Tue, Dec 13, 2016 at 3:24 PM, Sam Pegler <sam.pegler@infectiousmedia.com>
> wrote:
> 
>>> You can only check the offsets when there is an active member of the
>> consumer group.
>>
>> This was a bug [1] thats been fixed.  Thanks to Vincent Dautremont for
>> pointing this out to me a while ago.
>>
>> http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%
>> 3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.
>> gmail.com%3E
>>
>> __
>>
>> Sam Pegler
>>
>> PRODUCTION ENGINEER
>>
>> T. +44(0) 07 562 867 486
>>
>> <http://www.infectiousmedia.com/>
>> 3-7 Herbal Hill / London / EC1R 5EJ
>> www.infectiousmedia.com
>>
>> This email and any attachments are confidential and may also be privileged.
>> If you
>> are not the intended recipient, please notify the sender immediately, and
>> do not
>> disclose the contents to another person, use it for any purpose, or store,
>> or copy
>> the information in any medium. Please also destroy and delete the message
>> from
>> your computer.
>>
>>
>> On 13 December 2016 at 09:39, Damian Guy <damian.guy@gmail.com> wrote:
>>
>>> Hi Sachin
>>>
>>> That is correct. You can only check the offsets when there is an active
>>> member of the consumer group. In this case that would mean that you have
>> at
>>> least one instance of your streams application running.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Tue, 13 Dec 2016 at 06:58 Sachin Mittal <sjmittal@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I used the following command
>>>> bin\windows\kafka-consumer-groups.bat --bootstrap-server
>> localhost:9092
>>>> --describe --group test
>>>> and I get the following output
>>>>
>>>> Note: This will only show information about consumers that use the Java
>>>> consumer API (non-ZooKeeper-based consumers).
>>>>
>>>> Error: Consumer group 'test' has no active members.
>>>>
>>>> What does this mean.
>>>>
>>>> It means I can check the offset of consumer only when streams
>> applictaion
>>>> "test" is running.
>>>>
>>>> Thanks
>>>> Sachin
>>>>
>>>>
>>>> On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy <damian.guy@gmail.com>
>>> wrote:
>>>>
>>>>> Hi Sachin,
>>>>>
>>>>> You should use the kafka-consumer-groups.sh command. The
>>>>> ConsumerOffsetChecker is deprecated and is only for the old consumer.
>>>>>
>>>>> Thanks,
>>>>> Damian
>>>>>
>>>>> On Mon, 12 Dec 2016 at 14:32 Sachin Mittal <sjmittal@gmail.com>
>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I have a streams application running with application id test.
>>>>>> When I try to check consumer lag like you suggested I get the
>>> following
>>>>>> issue:
>>>>>>
>>>>>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
>> --zookeeper
>>>>>> localhost:2181 --group test
>>>>>> [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is
>>>>>> deprecated and will be dropped in releases following 0.9.0. Use
>>>>>> ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
>>>>>> SLF4J: Class path contains multiple SLF4J bindings.
>>>>>> SLF4J: Found binding in
>>>>>>
>>>>>> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
>>>>> libs/logback-classic-1.0.3.jar!/org/slf4j/impl/
>>> StaticLoggerBinder.class]
>>>>>> SLF4J: Found binding in
>>>>>>
>>>>>> [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/
>>>>> libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/
>>> StaticLoggerBinder.class]
>>>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
>> an
>>>>>> explanation.
>>>>>> SLF4J: Actual binding is of type
>>>>>> [ch.qos.logback.classic.selector.DefaultContextSelector]
>>>>>> Exiting due to: org.apache.zookeeper.KeeperException$
>>> NoNodeException:
>>>>>> KeeperErrorCode = NoNode for /consumers/test/owners.
>>>>>>
>>>>>> Please let me know where I may be going wrong.
>>>>>> I have the kafka logs set in folder
>>>>>> /data01/testuser/kafka-logs
>>>>>>
>>>>>> Under kafka-logs I see many folders with name something like
>>>>>> consumer_offsets_*
>>>>>>
>>>>>> I have the stream dir set in folder
>>>>>> /data01/testuser/kafka-streams/test
>>>>>>
>>>>>> Thanks
>>>>>> Sachin
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax <
>>>> matthias@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> It's basically just a consumer as any other. The application.id
>> is
>>>>> used
>>>>>>> as consumer group.id.
>>>>>>>
>>>>>>> So just use the available tools you do use to check consumer
lag.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 12/9/16 5:49 PM, Jon Yeargers wrote:
>>>>>>>> How would this be done?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> 


Mime
View raw message