kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Baugher <bjb...@gmail.com>
Subject Consumer members do not own any partitions in consumer group
Date Fri, 24 Apr 2015 16:13:11 GMT
Hi everyone,

We are running Kafka 0.8.1.1 with Storm. We wrote our own spout which uses
the high level consumer API. Our setup is to create 4 spouts per worker. If
your not familiar with Storm its basically 4 kafka consumers per java
process. This particular consumer group is interested in 20 topics and ~150
partitions. When we increased the parallelism to 6 workers and 24 consumers
we noticed certain consumers did not own any partitions in the group. These
consumers were on certain hosts. We see their ephemeral nodes in zookeeper
under their consumer group. We have also verified connectivity with kafka
from those nodes. I also found if I can get those workers/consumers to run
on the hosts with consumers that do own partitions they too will start
owning partitions.

I'm also finding nothing in our logs in Kafka or on the consumer side which
indicates any kind of problem.

Any suggestions on what to try? Is it possible to force a consumer
rebalance and who handles the partition assignment for a consumer group?

Bryan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message