kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Sheck <shec0...@gmail.com>
Subject Re: Closing a KafkaConsumer triggers a rebalance in KafkaConsumers subscribed to other topics. Is this the expected behavior?
Date Tue, 03 Apr 2018 16:43:43 GMT
I believe you have answered your own question.  Having the system enforce
which topics each consumer in a consumer group needs to consume messages
from may be restricting, and possibly expensive.  I have seen this brought
up a few times and discussed it with a colleague.  I am not clear as to
what the advantage is to have the members of a consumer group consume from
different topics.  I have seen instances where this was caused by a
misconfiguration, but are there cases where this would be done
intentionally?  If it doesn't exist already, maybe mismatched topics in a
consumer group should be alerted on?

On Tue, Apr 3, 2018 at 11:18 AM Gabriel Giussi <gabrielgiussi@gmail.com>
wrote:

> Kafka brokers version: 0.11.0.0
> Kafka client version: 0.11.0.2
>
> If we have two KafkaConsumer using the same group.id (running in the same
> process or in two different processes) and one of them is closed, it
> triggers a rebalance in the other KafkaConsumer even if they were
> subscribed to different topics.
> I suppose that brokers must be taking into account only the group.id for a
> rebalance and not the subscribed topics corresponding to the pair
> (group_id,member_id) of the LeaveGroupRequest but I'm wondering if this is
> the expected behavior or it's something that should be improved?
>
> I guess that is probably the first option to avoid a more complex rebalance
> in the broker and considering that the solution is very simple, i.e. just
> use different group ids for different KafkaConsumer that subscribe to
> different topics even if they are running in the same process.
>
> Thanks.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message