kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Barr <dave.b...@gmail.com>
Subject Re: Consumer re-design proposal
Date Wed, 20 Jun 2012 19:11:27 GMT
Yes, which is why I asked how discovery will work.

Proposal II reads a bit handwavy.

"On startup, the consumer is informed with a list of the brokers."

>From where?  From whom?  I have to provide some mechanism now to feed
this config to my consumers?  What happens when this changes?

Say I start up my producers and consumers with a static list of hosts
A,B,C.  Over time, I add host D, E, F.  D becomes the eventual new
coordinator.  But the clients have an in-memory config which knows
only of A, B, and C.  Now host D dies.  What happens (especially to
clients who have never been told about/connected to E or F)?

Also is this proposal going to address the ability to drain a broker?
Say I have a broker with old data that I want to keep around so you
can reconsume old data, but I don't want anyone producing to it
anymore.

--Dave
On Wed, Jun 20, 2012 at 7:48 AM, Jun Rao <junrao@gmail.com> wrote:
> Dave,
>
> Just to clarify. We are not removing ZK completely. The proposal is just
> trying to see if we can remove the ZK dependency on the consumer client. ZK
> will still be used at the broker.
>
> Thanks,
>
> Jun
>
> On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <dave.barr@gmail.com> wrote:
>
>> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <neha.narkhede@gmail.com>
>> wrote:
>> > One of the goals of thinning the Kafka consumer client is removing the
>> > zookeeper client from the consumer. Without this, Kafka consumer
>> > client would depend on the stability of a zookeeper client.
>>
>> If there's a stability issue with the zookeeper client, then that
>> should be addressed.
>>
>> ZK is a fine tool for service discovery and coordination.  It seems
>> like any new system that forced me, as a consumer, to use yet another
>> system to bootstrap and discover where my brokers are for a topic
>> would be a step backward.
>>
>> I'm curious, why, specifically, is removing ZK a design goal
>> (especially when it's such a core component of the broker)?  I think
>> of other projects, like HBase, which seem to have no issue with using
>> ZK in their client.
>>
>> --Dave
>>

Mime
View raw message