kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: consumer partition rebalancing
Date Wed, 11 Sep 2013 20:14:27 GMT
Correct - but since you wanted sticky allocation rebalancing wouldn't
really be necessary.



On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi <kamkasravi@yahoo.com> wrote:
> Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as suggested
below) does not join in partition rebalancing when another consumer in it's group joins or
a new broker joins or a new topic is created?
> ________________________________
>  From: Joel Koshy <jjkoshy.w@gmail.com>
> To: "users@kafka.apache.org" <users@kafka.apache.org>; Kam Kasravi <kamkasravi@yahoo.com>
> Sent: Tuesday, September 10, 2013 4:36 PM
> Subject: Re: consumer partition rebalancing
>>         * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a
topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be
the only reader of that partition:
>>         * Are there ways to scale partition consumption if C(i) cannot keep up?
> To some degree - by either reducing the number of streams that C(i) is
> configured with; or by removing it altogether.
>>         * A way to designate particular consumers should consume particular partitions
based on consumer capability (similar to container allocation in YARN)?
>>         * A way to designate sticky consumer / partition allocation so that a new
consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular
partitions during rebalancing?
> Right now, no. I think what you may be effectively suggesting is to
> support some form of pluggable logic for partition ownership which
> should work as long as it is consistent across all consumer instances.
> However, a designated assignment list could be achieved by using
> SimpleConsumer - although you would then need to write a little code
> to react to leader changes. (We currently don't have a
> intermediate-level consumer - i.e., a consumer that can be designated
> to consume from a given partition and automatically react to leader
> changes.)
> Joel

View raw message