kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yury Ruchin <yuri.ruc...@gmail.com>
Subject Kafka 0.8's VerifyConsumerRebalance reports an error
Date Tue, 24 Jun 2014 07:02:32 GMT
Hi,

I've run into the following problem. I try to read from a 50-partition
Kafka topic using high level consumer with 8 streams. I'm using 8-thread
pool, each thread handling one stream. After a short time, the threads
reading from the stream stop reading. Lag between topic latest offset and
the consumer constantly increases as new messages come in.

I looked into ZK:  /consumers/<consumer_name>/owners/<topic_name> and see a
list of znodes corresponding to the full list of partitions: [1, 2, 3,
...]. When I do zk get on e. g.
/consumers/<consumer_name>/owners/<topic_name>/1 - I see a valid consumer
name corresponding to Kafka client logs, e. g.
<consumer_name>_<node_name>-1403049600000-abc12345-0. However, when I run
the VerifyConsumerRebalance tool, I see the following:

No owner for partition [<topic_name>,1]
(kafka.tools.VerifyConsumerRebalance$)

No owner for partition [<topic_name>,2]
(kafka.tools.VerifyConsumerRebalance$)

...

No owner for partition [<topic_name>,50]
(kafka.tools.VerifyConsumerRebalance$)

According to this output, no partition has owner, which seemingly
contradicts to what I see in ZK.

What would cause such a problem and how can I troubleshoot it further?

Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message