kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Baugher <bjb...@gmail.com>
Subject Round Robin Partition Assignment
Date Tue, 05 May 2015 21:20:05 GMT
Hi everyone,

We recently switched to round robin partition assignment after we noticed
that range partition assignment (default) will only make use of the first X
consumers were X is the number of partitions for a topic our consumers are
interested in. We then noticed the caveat in round robin,

"Round-robin assignment is permitted only if: (a) Every topic has the same
number of streams within a consumer instance (b) The set of subscribed
topics is identical for every consumer instance within the group."

We tried this out and found if all consumers don't agree on topic
subscription they basically stop consuming until things get figured out.
This is bad for us since our consumers change their topic subscription
based on config they load from a REST service periodically.

Is there something we can do on our side to avoid this? The best thing I
can think of is to try and use something like zookeeper to coordinate
changing the topic filters.

Would it be possible to see the round robin assignment updated to not
require identical topic subscriptions?

Bryan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message