kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlastimil Menčík <men...@avast.com>
Subject A more reliable way of consumer leaving group when it is closed?
Date Wed, 19 Oct 2016 09:14:08 GMT
Hi.



I’ve run into a problem with Kafka consumers not leaving their consumer
group cleanly when the application restarts.

Out of 9 topics that (with a consumer for each) it seems that every time at
least 2 or 3 do not leave the group cleanly so when the application starts
these consumers start consuming only after session.timeout.ms expires and
the group re-balances.

When debugging this I’ve enabled logging from GroupCoordinator in the
broker and I’ve really observed that not all of LeaveGroupRequests were
received.



I’ve looked at the code in KafkaConsumer.close() and the effort to send the
LeaveGroupRequest really is minimal (and the comments suggest this is the
intention). So if the network client cannot send the request immediately,
it is not sent at all.



But I think that the amount of effort put into sending LeaveGroupRequest
should be the decision of the consumer’s user, not the library itself.

IMHO it would be enough if the poll timeout when sending LeaveGroupRequest
was configurable.



We’re using Kafka 0.10.0.1. I’ve looked at the KafkaConsumer.close() code
in the master branch and it looks like is has changed quite a bit when
compared to 0.10.0.1 but the LeaveGroupRequest sending effort has not
changed IMHO.



Vlastimil

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message