kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Filipiak <Jan.Filip...@trivago.com>
Subject Re: Kafka Consumer Offsets unavailable during rebalancing
Date Tue, 13 Feb 2018 11:50:40 GMT
I would encourage you todo so.
I also think its not reasonable behavior

On 13.02.2018 11:28, Wouter Bancken wrote:
> We have upgraded our Kafka version as an attempt to solve this issue.
> However, the issue is still present in Kafka 1.0.0.
>
> Can I log a bug for this in JIRA?
>
> Wouter
>
> On 5 February 2018 at 09:22, Wouter Bancken <wouter.bancken@aca-it.be>
> wrote:
>
>> The consumers in consumer group 'X' do not have a regex subscription
>> matching the newly created topic 'C'. They simply subscribe with
>> the subscribe(java.util.Collection<java.lang.String> topics) method on
>> topics 'A' and 'B'.
>>
>> Shouldn't the consumer group have a different state from "Stable" during a
>> rebalancing regardless of the cause? How else can we determine the consumer
>> lag of the group during the rebalancing?
>>
>> Best regards,
>> Wouter
>>
>> Have a look at our brand NEW job website: jobs.aca-it.be !
>>
>>
>> *ACA IT-Solutions NV*
>> *HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt
>> T +32(0)11 26 50 10 | F +32(0)11 26 50 11
>> www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook
>> <http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> |
>> Linkedin <http://www.linkedin.com/company/6524>
>>
>> On 5 February 2018 at 00:13, Hans Jespersen <hans@confluent.io> wrote:
>>
>>> Do the consumers in consumer group ‘X’ have a regex subscription that
>>> matches the newly created topic ‘C’?
>>>
>>> If they do then they will only discover this new topic once their ‘
>>> metadata.max.age.ms’  metadata refresh interval has passed, which
>>> defaults to 5 minutes.
>>>
>>> metadata.max.age.ms     The period of time in milliseconds after which
>>> we force a refresh of metadata even if we haven't seen any partition
>>> leadership changes to proactively discover any new brokers or partitions
>>> -hans
>>>
>>>
>>>> On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wouter.bancken@aca-it.be>
>>> wrote:
>>>> Hi Hans,
>>>>
>>>> Thanks for the response!
>>>>
>>>> However, I get this result for all topics, not just for the newly
>>> created
>>>> topic.
>>>>
>>>> Situation sketch:
>>>> 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
>>>> partition assignments and lag information. Consumer group 'X' is
>>> "Stable".
>>>> 2a. Topic 'C' is (being) created.
>>>> 2b. During this creation, I do not have a partition assignment for
>>> consumer
>>>> group 'X' for topics 'A' and 'B' but the consumer group is still
>>> "Stable".
>>>> 3. A second later: I have a partition assignment for consumer group 'X'
>>> for
>>>> topics 'A' and 'B' again and the consumer group is still "Stable".
>>>>
>>>> I expected the state of consumer group 'X' during step 2b to be
>>>> "PreparingRebalance" or "AwaitingSync".
>>>>
>>>> Best regards,
>>>> Wouter
>>>>
>>>>> On 4 February 2018 at 21:25, Hans Jespersen <hans@confluent.io>
wrote:
>>>>>
>>>>> I believe this is expected behavior.
>>>>>
>>>>> If there are no subscriptions to a new topic, and therefor no partition
>>>>> assignments, and definitely no committed offsets, then lag is an
>>> undefined
>>>>> concept. When the consumers subscribe to this new topic they may chose
>>> to
>>>>> start at the beginning or end of the commit log so the lag cannot be
>>>>> predicted in advance.
>>>>>
>>>>> -hans
>>>>>
>>>>>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.bancken@aca-it.be
>>>>> wrote:
>>>>>> Can anyone clarify if this is a bug in Kafka or the expected behavior?
>>>>>>
>>>>>> Best regards,
>>>>>> Wouter
>>>>>>
>>>>>>
>>>>>> On 30 January 2018 at 21:04, Wouter Bancken <wouter.bancken@aca-it.be
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying to write an external tool to monitor consumer lag
on
>>> Apache
>>>>>>> Kafka.
>>>>>>>
>>>>>>> For this purpose, I'm using the kafka-consumer-groups tool to
fetch
>>> the
>>>>>>> consumer offsets.
>>>>>>>
>>>>>>> When using this tool, partition assignments seem to be unavailable
>>>>>>> temporarily during the creation of a new topic even if the consumer
>>>>> group
>>>>>>> has no subscription on this new topic. This seems to match the
>>>>>>> documentation
>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/
>>>>> Kafka+Client-side+Assignment+Proposal>
>>>>>>> saying *"Topic metadata changes which have no impact on subscriptions
>>>>>>> cause resync"*.
>>>>>>>
>>>>>>> However, when this occurs I'd expect the state of the consumer
to be
>>>>>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
>>>>>>>
>>>>>>> Is this a bug in the tooling or is there a different way to obtain
>>> the
>>>>>>> correct offsets for a consumer group during a rebalance?
>>>>>>>
>>>>>>> I'm using Kafka 10.2.1 but I haven't found any related issues
in
>>> recent
>>>>>>> changelogs.
>>>>>>> Best regards,
>>>>>>> Wouter
>>>>>>>
>>


Mime
View raw message