kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From C├ędric Chantepie <c.chante...@yahoo.fr.INVALID>
Subject Heartbeat failed w/ multi-group/consumers
Date Wed, 17 May 2017 23:59:06 GMT
Hi,

I have a test app using Java lib for consumers with Kafka 0.10, using Kafka storage for offset.

This app is managing 190 consumers, accross 19 different consumer group, against 12 distinct
topics (details bellow).

When one app instance is starting, with 40 partitions per topic, it takes ~1m to get stable
assignments on topic partitions for all the workers: long time but could be understandable
...

When two instances of the app are started (same kafka server setup), it goes through an infinite
rebalancing loop, not consuming anything.

This test app is just printing data for now, no long processing. I've tried several settings
(session.timeout, heartbeat, client.id, ...), but I cannot fix that.

Is there a know issue related to such case?

For more details, this app is simulating consumpting by several kinds of worker/processor,
with one distinct consumer group per kind.

Some different kinds of worker, and so different consumer groups, could subscribe to a same
topic.

The app is so using 19 distinct consumer groups (1 per simulated worker kind), against 12
different topics.

Moreover, per each kind of worker, 10 consumer instances are created and subscribe using the
same consumer group to the appropriate topic.
Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message