kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiangjie Qin <j...@linkedin.com.INVALID>
Subject Re: Kafka 0.8.3 - New Consumer - user implemented partition.assignment.strategies?
Date Thu, 18 Jun 2015 23:37:19 GMT
Hi Olof,

I am just wondering what is the benefit of rebalancing with minimal number
of reassignments here?

I am asking this because in new consumer, the rebalance actually is quite
cheap on the consumer side - just updating a topic partition set. That
means the actually rebalance time on consumer side is almost ignorable no
matter how many partitions are reassigned.

Is it because the consumer has some sort of sticking partition
requirements? If that is the case,  that seems need an immutable partition
assignment policy.

Just curious about the motivation behind this.

Thanks,

Jiangjie (Becket) Qin


On 6/11/15, 9:30 AM, "Johansson, Olof" <Olof.Johansson@thingworx.com>
wrote:

>Thanks Guozhang,
>
>I agree that it seems to be a common reassignment strategy that should be
>one of the pre-defined strategies. Do you have a Jira ticket for this
>specific case, and/or a Jira ticket to add user defined
>partitions.assignment.strategies that I can watch?
>
>/ Olof
>
>On 10/06/2015 14:35, "Guozhang Wang" <wangguoz@gmail.com> wrote:
>
>>Hi Olof,
>>
>>Yes we have plans to allow user defined partitions.assignment strategy to
>>pass in to the consumer coordinator; I am not sure if this feature will
>>not
>>be available in the first release of the new consumer in 0.8.3 though.
>>Currently users still have to choose one from the server-defined strategy
>>upon registering themselves.
>>
>>On the other hand, I think "rebalance with a minimal number of
>>reassignment" should be quite a common reassignment strategy, and I think
>>it is possible to just add it into the server-defined strategies list.
>>
>>Guozhang
>>
>>
>>On Wed, Jun 10, 2015 at 9:32 AM, Johansson, Olof <
>>Olof.Johansson@thingworx.com> wrote:
>>
>>> Is it possible to have a consumer rebalancing partition assignment
>>> strategy that will rebalance with a minimal number of reassignments?
>>>
>>> Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to
>>> define my own partition assignment strategy:
>>> "partition.assignment.strategies - may take a comma separated list of
>>> properties that map the strategy's friendly name to to the class that
>>> implements the strategy. This is used for any strategy implemented by
>>>the
>>> user and released to the Kafka cluster. By default, Kafka will include
>>>a
>>> set of strategies that can be used by the consumer."
>>>
>>> Is there a Jira ticket that tracks adding user defined
>>> partitions.assignment.strategies? In the latest source, range, and
>>> roundrobin are still the only possible values (hard-coded).
>>>
>>> I assume that any user implemented strategy would have to implement the
>>> PartitionAssignor trait. If so, by naively looking at the 0.8.3 source,
>>>a
>>> strategy that should do a minimal number of partition reassignments
>>>would
>>> need the ConsumerMetaData. That's not currently available in the
>>> PartitionAssignor contract - assign(topicsPerConsumer,
>>>partitionsPerTopic).
>>> Have there been any discussion to change the contract to pass
>>> ConsumerMetaData?
>>>
>>
>>
>>
>>-- 
>>-- Guozhang
>


Mime
View raw message