kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Baugher <bjb...@gmail.com>
Subject Re: Consumer members do not own any partitions in consumer group
Date Fri, 24 Apr 2015 20:26:57 GMT
Managed to figure this one out myself. This is due to the range partition
assignment in 0.8.1.1 and the fact each of our topics has 8 partitions so
only the first 8 consumers get assigned anything. Looks like 0.8.2.0 has a
round robin assignment which is what we want.

On Fri, Apr 24, 2015 at 11:13 AM Bryan Baugher <bjbq4d@gmail.com> wrote:

> Hi everyone,
>
> We are running Kafka 0.8.1.1 with Storm. We wrote our own spout which uses
> the high level consumer API. Our setup is to create 4 spouts per worker. If
> your not familiar with Storm its basically 4 kafka consumers per java
> process. This particular consumer group is interested in 20 topics and ~150
> partitions. When we increased the parallelism to 6 workers and 24 consumers
> we noticed certain consumers did not own any partitions in the group. These
> consumers were on certain hosts. We see their ephemeral nodes in zookeeper
> under their consumer group. We have also verified connectivity with kafka
> from those nodes. I also found if I can get those workers/consumers to run
> on the hosts with consumers that do own partitions they too will start
> owning partitions.
>
> I'm also finding nothing in our logs in Kafka or on the consumer side
> which indicates any kind of problem.
>
> Any suggestions on what to try? Is it possible to force a consumer
> rebalance and who handles the partition assignment for a consumer group?
>
> Bryan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message