kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhimanyu Nagrath <abhimanyunagr...@gmail.com>
Subject Re: Heartbeat failed w/ multi-group/consumers
Date Thu, 18 May 2017 03:33:13 GMT
I am also facing the same issue .


Regards,
Abhimanyu

On Thu, May 18, 2017 at 5:29 AM, C├ędric Chantepie <
c.chantepie@yahoo.fr.invalid> wrote:

> Hi,
>
> I have a test app using Java lib for consumers with Kafka 0.10, using
> Kafka storage for offset.
>
> This app is managing 190 consumers, accross 19 different consumer group,
> against 12 distinct topics (details bellow).
>
> When one app instance is starting, with 40 partitions per topic, it takes
> ~1m to get stable assignments on topic partitions for all the workers: long
> time but could be understandable ...
>
> When two instances of the app are started (same kafka server setup), it
> goes through an infinite rebalancing loop, not consuming anything.
>
> This test app is just printing data for now, no long processing. I've
> tried several settings (session.timeout, heartbeat, client.id, ...), but
> I cannot fix that.
>
> Is there a know issue related to such case?
>
> For more details, this app is simulating consumpting by several kinds of
> worker/processor, with one distinct consumer group per kind.
>
> Some different kinds of worker, and so different consumer groups, could
> subscribe to a same topic.
>
> The app is so using 19 distinct consumer groups (1 per simulated worker
> kind), against 12 different topics.
>
> Moreover, per each kind of worker, 10 consumer instances are created and
> subscribe using the same consumer group to the appropriate topic.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message