kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johansson, Olof" <Olof.Johans...@thingworx.com>
Subject Re: Kafka 0.8.3 - New Consumer - user implemented partition.assignment.strategies?
Date Thu, 11 Jun 2015 16:30:05 GMT
Thanks Guozhang,

I agree that it seems to be a common reassignment strategy that should be
one of the pre-defined strategies. Do you have a Jira ticket for this
specific case, and/or a Jira ticket to add user defined
partitions.assignment.strategies that I can watch?

/ Olof

On 10/06/2015 14:35, "Guozhang Wang" <wangguoz@gmail.com> wrote:

>Hi Olof,
>
>Yes we have plans to allow user defined partitions.assignment strategy to
>pass in to the consumer coordinator; I am not sure if this feature will
>not
>be available in the first release of the new consumer in 0.8.3 though.
>Currently users still have to choose one from the server-defined strategy
>upon registering themselves.
>
>On the other hand, I think "rebalance with a minimal number of
>reassignment" should be quite a common reassignment strategy, and I think
>it is possible to just add it into the server-defined strategies list.
>
>Guozhang
>
>
>On Wed, Jun 10, 2015 at 9:32 AM, Johansson, Olof <
>Olof.Johansson@thingworx.com> wrote:
>
>> Is it possible to have a consumer rebalancing partition assignment
>> strategy that will rebalance with a minimal number of reassignments?
>>
>> Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to
>> define my own partition assignment strategy:
>> "partition.assignment.strategies - may take a comma separated list of
>> properties that map the strategy's friendly name to to the class that
>> implements the strategy. This is used for any strategy implemented by
>>the
>> user and released to the Kafka cluster. By default, Kafka will include a
>> set of strategies that can be used by the consumer."
>>
>> Is there a Jira ticket that tracks adding user defined
>> partitions.assignment.strategies? In the latest source, range, and
>> roundrobin are still the only possible values (hard-coded).
>>
>> I assume that any user implemented strategy would have to implement the
>> PartitionAssignor trait. If so, by naively looking at the 0.8.3 source,
>>a
>> strategy that should do a minimal number of partition reassignments
>>would
>> need the ConsumerMetaData. That's not currently available in the
>> PartitionAssignor contract - assign(topicsPerConsumer,
>>partitionsPerTopic).
>> Have there been any discussion to change the contract to pass
>> ConsumerMetaData?
>>
>
>
>
>-- 
>-- Guozhang


Mime
View raw message