kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Consumer re-design proposal
Date Wed, 20 Jun 2012 17:01:23 GMT
The proposal is to move the rebalancing logic out of the consumer to
the broker side. The consumer client merely connects to the consumer
co-ordinator (on the Kafka brokers) and fetches data from the assigned
set of partitions. We have a high level design here -
https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination#CentralConsumerCoordination-ProposalII

Thanks,
Neha

On Wed, Jun 20, 2012 at 9:36 AM, Evan Chan <ev@ooyala.com> wrote:
> How would you accomplish automatic rebalancing without ZK?
>
> On Wed, Jun 20, 2012 at 8:25 AM, Taylor Gautier <tgautier@tagged.com> wrote:
>
>> Fwiw I think this is the right move. We don't use ZK in our Kafka
>> installation.
>>
>> One reason is that we do not want or need the complexity of having the
>> broker distribution in the clients.
>>
>> Moving towards this design in the core would help our installation be
>> more "standard".
>>
>>
>>
>> On Jun 20, 2012, at 7:48 AM, Jun Rao <junrao@gmail.com> wrote:
>>
>> > Dave,
>> >
>> > Just to clarify. We are not removing ZK completely. The proposal is just
>> > trying to see if we can remove the ZK dependency on the consumer client.
>> ZK
>> > will still be used at the broker.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <dave.barr@gmail.com> wrote:
>> >
>> >> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <
>> neha.narkhede@gmail.com>
>> >> wrote:
>> >>> One of the goals of thinning the Kafka consumer client is removing the
>> >>> zookeeper client from the consumer. Without this, Kafka consumer
>> >>> client would depend on the stability of a zookeeper client.
>> >>
>> >> If there's a stability issue with the zookeeper client, then that
>> >> should be addressed.
>> >>
>> >> ZK is a fine tool for service discovery and coordination.  It seems
>> >> like any new system that forced me, as a consumer, to use yet another
>> >> system to bootstrap and discover where my brokers are for a topic
>> >> would be a step backward.
>> >>
>> >> I'm curious, why, specifically, is removing ZK a design goal
>> >> (especially when it's such a core component of the broker)?  I think
>> >> of other projects, like HBase, which seem to have no issue with using
>> >> ZK in their client.
>> >>
>> >> --Dave
>> >>
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>

Mime
View raw message