Sorry, I should say only partition 1 had something at first, then zero:
TomsiMac:betwaveserver tomdearman$ /Users/tomdearman/software/kafka_2.110.10.0.0/bin/kafkaconsumergroups.sh
newconsumer bootstrapserver localhost:9092 describe group voidbridgeoneworksdummy
GROUP TOPIC PARTITION CURRENTOFFSET LOGENDOFFSET
LAG OWNER
voidbridgeoneworksdummy integrationoneworksdummy 2 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy3_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 7 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy3_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 12 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy3_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 17 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy3_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 4 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy5_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 9 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy5_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 14 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy5_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 19 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy5_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 1 3 3
0 integrationoneworksdummyvoidbridgeoneworksdummy2_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 6 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy2_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 11 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy2_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 16 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy2_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 3 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy4_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 8 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy4_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 13 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy4_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 18 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy4_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 0 10 0
10 integrationoneworksdummyvoidbridgeoneworksdummy1_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 5 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy1_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 10 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy1_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 15 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy1_/10.100.0.113
TomsiMac:betwaveserver tomdearman$ /Users/tomdearman/software/kafka_2.110.10.0.0/bin/kafkaconsumergroups.sh
newconsumer bootstrapserver localhost:9092 describe group voidbridgeoneworksdummy
GROUP TOPIC PARTITION CURRENTOFFSET LOGENDOFFSET
LAG OWNER
voidbridgeoneworksdummy integrationoneworksdummy 2 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy3_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 7 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy3_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 12 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy3_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 17 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy3_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 4 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy5_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 9 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy5_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 14 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy5_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 19 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy5_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 1 3 3
0 integrationoneworksdummyvoidbridgeoneworksdummy2_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 6 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy2_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 11 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy2_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 16 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy2_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 3 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy4_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 8 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy4_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 13 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy4_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 18 unknown 0
unknown integrationoneworksdummyvoidbridgeoneworksdummy4_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 0 1 1
0 integrationoneworksdummyvoidbridgeoneworksdummy1_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 5 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy1_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 10 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy1_/10.100.0.113
voidbridgeoneworksdummy integrationoneworksdummy 15 0 0
0 integrationoneworksdummyvoidbridgeoneworksdummy1_/10.100.0.113
> On 8 Jul 2016, at 17:20, Tom Dearman <tom.dearman@gmail.com> wrote:
>
> When you say ‘for the first partition’ do you literally mean partition zero, or you
mean any partition. It is true that when I had only 1 user there were only messages on partition
15 but the second user happened to go to partition zero. Is it the case that partition zero
must have a consumer commit?
>
>> On 8 Jul 2016, at 17:16, Todd Palino <tpalino@gmail.com> wrote:
>>
>> If you open up an issue on the project, I'd be happy to dig into this in
>> more detail if needed. Excluding the ZK offset checking, Burrow doesn't
>> enumerate consumer groups  it learns about them from offset commits. It
>> sounds like maybe your consumer had not committed offsets for the first
>> partition (at least not after Burrow was started).
>>
>> Todd
>>
>> On Friday, July 8, 2016, Tom Dearman <tom.dearman@gmail.com> wrote:
>>
>>> Todd,
>>>
>>> Thanks for that I am taking a look.
>>>
>>> Is there a bug whereby if you only have a couple of messages on a topic,
>>> both with the same key, that burrow doesn’t return correct info. I was
>>> finding that http://localhost:8100/v2/kafka/betwave/consumer <
>>> http://localhost:8100/v2/kafka/betwave/consumer> was returning a message
>>> with empty consumers until I put on another message with a different key,
>>> i.e. a minimum of 2 partitions with something in them. I know this is not
>>> very like production, but on my local this I was only testing with one user
>>> so get just one partition filled.
>>>
>>> Tom
>>>> On 6 Jul 2016, at 18:08, Todd Palino <tpalino@gmail.com <javascript:;>>
>>> wrote:
>>>>
>>>> Yeah, I've written dissertations at this point on why MaxLag is flawed.
>>> We
>>>> also used to use the offset checker tool, and later something similar
>>> that
>>>> was a little easier to slot into our monitoring systems. Problems with
>>> all
>>>> of these is why I wrote Burrow (https://github.com/linkedin/Burrow)
>>>>
>>>> For more details, you can also check out my blog post on the release:
>>>>
>>> https://engineering.linkedin.com/apachekafka/burrowkafkaconsumermonitoringreinvented
>>>>
>>>> Todd
>>>>
>>>> On Wednesday, July 6, 2016, Tom Dearman <tom.dearman@gmail.com
>>> <javascript:;>> wrote:
>>>>
>>>>> I recently had a problem on my production which I believe was a
>>>>> manifestation of the issue kafka2978 (Topic partition is not sometimes
>>>>> consumed after rebalancing of consumer group), this is fixed in 0.9.0.1
>>> and
>>>>> we will upgrade our client soon. However, it made me realise that I
>>> didn’t
>>>>> have any monitoring set up on this. The only thing I can find as a
>>> metric
>>>>> is the
>>>>>
>>> kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([.\w]+),
>>>>> which, if I understand correctly, is the max lag of any partition that
>>> that
>>>>> particular consumer is consuming.
>>>>> 1. If I had been monitoring this, and if my consumer was suffering from
>>>>> the issue in kafka2978, would I actually have been alerted, i.e. since
>>> the
>>>>> consumer would think it is consuming correctly would it not have updated
>>>>> the metric.
>>>>> 2. There is another way to see offset lag using the command
>>>>> /usr/bin/kafkaconsumergroups newconsumer bootstrapserver
>>>>> 10.10.1.61:9092 describe —group consumer_group_name and parsing the
>>>>> response. Is it safe or advisable to do this? I like the fact that
it
>>>>> tells me each partition lag, although it is also not available if no
>>>>> consumer from the group is currently consuming.
>>>>> 3. Is there a better way of doing this?
>>>>
>>>>
>>>>
>>>> 
>>>> *Todd Palino*
>>>> Staff Site Reliability Engineer
>>>> Data Infrastructure Streaming
>>>>
>>>>
>>>>
>>>> linkedin.com/in/toddpalino
>>>
>>>
>>
>> 
>> *Todd Palino*
>> Staff Site Reliability Engineer
>> Data Infrastructure Streaming
>>
>>
>>
>> linkedin.com/in/toddpalino
>
