kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johansson, Olof" <Olof.Johans...@thingworx.com>
Subject Kafka 0.8.3 - New Consumer - user implemented partition.assignment.strategies?
Date Wed, 10 Jun 2015 16:32:10 GMT
Is it possible to have a consumer rebalancing partition assignment strategy that will rebalance
with a minimal number of reassignments?

Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to define my own partition
assignment strategy: "partition.assignment.strategies - may take a comma separated list of
properties that map the strategy's friendly name to to the class that implements the strategy.
This is used for any strategy implemented by the user and released to the Kafka cluster. By
default, Kafka will include a set of strategies that can be used by the consumer."

Is there a Jira ticket that tracks adding user defined partitions.assignment.strategies? In
the latest source, range, and roundrobin are still the only possible values (hard-coded).

I assume that any user implemented strategy would have to implement the PartitionAssignor
trait. If so, by naively looking at the 0.8.3 source, a strategy that should do a minimal
number of partition reassignments would need the ConsumerMetaData. That's not currently available
in the PartitionAssignor contract - assign(topicsPerConsumer, partitionsPerTopic). Have there
been any discussion to change the contract to pass ConsumerMetaData?

View raw message