kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Dearman <tom.dear...@gmail.com>
Subject Re: Monitoring offset lag
Date Fri, 08 Jul 2016 16:20:36 GMT
When you say ‘for the first partition’ do you literally mean partition zero, or you mean
any partition.  It is true that when I had only 1 user there were only messages on partition
15 but the second user happened to go to partition zero.  Is it the case that partition zero
must have a consumer commit?

> On 8 Jul 2016, at 17:16, Todd Palino <tpalino@gmail.com> wrote:
> 
> If you open up an issue on the project, I'd be happy to dig into this in
> more detail if needed. Excluding the ZK offset checking, Burrow doesn't
> enumerate consumer groups - it learns about them from offset commits. It
> sounds like maybe your consumer had not committed offsets for the first
> partition (at least not after Burrow was started).
> 
> -Todd
> 
> On Friday, July 8, 2016, Tom Dearman <tom.dearman@gmail.com> wrote:
> 
>> Todd,
>> 
>> Thanks for that I am taking a look.
>> 
>> Is there a bug whereby if you only have a couple of messages on a topic,
>> both with the same key, that burrow doesn’t return correct info.  I was
>> finding that http://localhost:8100/v2/kafka/betwave/consumer <
>> http://localhost:8100/v2/kafka/betwave/consumer> was returning a message
>> with empty consumers until I put on another message with a different key,
>> i.e. a minimum of 2 partitions with something in them.  I know this is not
>> very like production, but on my local this I was only testing with one user
>> so get just one partition filled.
>> 
>> Tom
>>> On 6 Jul 2016, at 18:08, Todd Palino <tpalino@gmail.com <javascript:;>>
>> wrote:
>>> 
>>> Yeah, I've written dissertations at this point on why MaxLag is flawed.
>> We
>>> also used to use the offset checker tool, and later something similar
>> that
>>> was a little easier to slot into our monitoring systems. Problems with
>> all
>>> of these is why I wrote Burrow (https://github.com/linkedin/Burrow)
>>> 
>>> For more details, you can also check out my blog post on the release:
>>> 
>> https://engineering.linkedin.com/apache-kafka/burrow-kafka-consumer-monitoring-reinvented
>>> 
>>> -Todd
>>> 
>>> On Wednesday, July 6, 2016, Tom Dearman <tom.dearman@gmail.com
>> <javascript:;>> wrote:
>>> 
>>>> I recently had a problem on my production which I believe was a
>>>> manifestation of the issue kafka-2978 (Topic partition is not sometimes
>>>> consumed after rebalancing of consumer group), this is fixed in 0.9.0.1
>> and
>>>> we will upgrade our client soon.  However, it made me realise that I
>> didn’t
>>>> have any monitoring set up on this.  The only thing I can find as a
>> metric
>>>> is the
>>>> 
>> kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+),
>>>> which, if I understand correctly, is the max lag of any partition that
>> that
>>>> particular consumer is consuming.
>>>> 1. If I had been monitoring this, and if my consumer was suffering from
>>>> the issue in kafka-2978, would I actually have been alerted, i.e. since
>> the
>>>> consumer would think it is consuming correctly would it not have updated
>>>> the metric.
>>>> 2. There is another way to see offset lag using the command
>>>> /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server
>>>> 10.10.1.61:9092 --describe —group consumer_group_name and parsing the
>>>> response.  Is it safe or advisable to do this?  I like the fact that it
>>>> tells me each partition lag, although it is also not available if no
>>>> consumer from the group is currently consuming.
>>>> 3. Is there a better way of doing this?
>>> 
>>> 
>>> 
>>> --
>>> *Todd Palino*
>>> Staff Site Reliability Engineer
>>> Data Infrastructure Streaming
>>> 
>>> 
>>> 
>>> linkedin.com/in/toddpalino
>> 
>> 
> 
> -- 
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
> 
> 
> 
> linkedin.com/in/toddpalino


Mime
View raw message