kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Kafka 0.8.3 - New Consumer - user implemented partition.assignment.strategies?
Date Wed, 10 Jun 2015 18:35:03 GMT
Hi Olof,

Yes we have plans to allow user defined partitions.assignment strategy to
pass in to the consumer coordinator; I am not sure if this feature will not
be available in the first release of the new consumer in 0.8.3 though.
Currently users still have to choose one from the server-defined strategy
upon registering themselves.

On the other hand, I think "rebalance with a minimal number of
reassignment" should be quite a common reassignment strategy, and I think
it is possible to just add it into the server-defined strategies list.

Guozhang


On Wed, Jun 10, 2015 at 9:32 AM, Johansson, Olof <
Olof.Johansson@thingworx.com> wrote:

> Is it possible to have a consumer rebalancing partition assignment
> strategy that will rebalance with a minimal number of reassignments?
>
> Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to
> define my own partition assignment strategy:
> "partition.assignment.strategies - may take a comma separated list of
> properties that map the strategy's friendly name to to the class that
> implements the strategy. This is used for any strategy implemented by the
> user and released to the Kafka cluster. By default, Kafka will include a
> set of strategies that can be used by the consumer."
>
> Is there a Jira ticket that tracks adding user defined
> partitions.assignment.strategies? In the latest source, range, and
> roundrobin are still the only possible values (hard-coded).
>
> I assume that any user implemented strategy would have to implement the
> PartitionAssignor trait. If so, by naively looking at the 0.8.3 source, a
> strategy that should do a minimal number of partition reassignments would
> need the ConsumerMetaData. That's not currently available in the
> PartitionAssignor contract - assign(topicsPerConsumer, partitionsPerTopic).
> Have there been any discussion to change the contract to pass
> ConsumerMetaData?
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message