kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Multi-topic consumer configuration
Date Thu, 22 Dec 2016 03:46:35 GMT
Do each of your topics have only 1 partition? The default partition
assignor is org.apache.kafka.clients.consumer.RangeAssignor which will give
the behavior you describe if each topic only has a single partition.

You probably want to switch it to RoundRobinAssignor which guarantees a
more even distribution of topic partitions to consumers.

-Ewen

On Wed, Dec 21, 2016 at 10:15 AM, Robert Quinlivan <rquinlivan@signal.co>
wrote:

> Hello,
>
> We are using a multi-topic approach for partitioning data. Each topic will
> follow a pattern naming convention. We want to design our consumer
> configuration so that a slow topic will not block a fast topic, as each
> topic will consume at a different rate.
>
> However I am seeing an issue where all topics matching the pattern are
> assigned to a single consumer. When there are several consumers in the
> group with a single topic with many partitions, the partitions are
> automatically distributed among the available consumers in the group. I
> expected that calling KafkaConsumer#subscribe(Pattern,
> ConsumerRebalanceListener) would follow a similar behavior by distributing
> the assigned topics among all consumers in the group.
>
> Is this not the case? What is the expected behavior and how would you
> recommend implementing this design?
>
> Thank you
>
> --
> Robert Quinlivan
> Software Engineer, Signal
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message