kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boyang Chen <reluctanthero...@gmail.com>
Subject Re: (Re-)joining group for a longer time
Date Sat, 03 Aug 2019 16:36:02 GMT
Hey Kamesh,

thank you for the question. Could you also check the broker side log to see
if the group is forming generations properly? Information we have for now
is a bit hard to tell what's going on. Also since you have upgraded to 2.3,
during incremental rebalancing you will experience 2 rebalance in a row but
won't revoke/assign tasks unless necessary, could you verify that for old
connectors their partitions are not getting revoked during the first
rebalance?

Boyang



On Sat, Aug 3, 2019 at 1:40 AM Kamesh <kam.iitkgp@gmail.com> wrote:

> Hi,
>  I am using Kafka connect cluster for writing data to S3. I have observed,
> whenever I add a new connector or update the config of an existing
> connector, I think group balancing is happening and it is affecting all the
> existing connectors. Rebalancing is happening for all of the existing
> connectors also. All of my log files are filled with the following log
> messages
>
> *[2019-08-03 08:28:28,668] INFO [Consumer clientId=connector-consumer-xxx,
> groupId=connect-xxx] (Re-)joining group
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)*
>
>  This rebalancing is taking a very long time, and sometimes not evening
> completing. Any pointers on this?
>
> I am using Kafka connect *2.3.0, *and this version supports incremental
> rebalancing and it should not affect existing connectors. Am I
> missing something?
>
> I have already tuned *session.timeout.ms <http://session.timeout.ms>*
> and *max.poll.interval.ms
> <http://max.poll.interval.ms>* configs and increased their values as
> suggested in the community.
>
>
> Thanks & Regards
> Kamesh.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message