kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Gustafson <ja...@confluent.io>
Subject Re: The motive to detach new consumer and zookeeper
Date Mon, 07 Dec 2015 17:55:39 GMT
Hi Li,

I think reducing the client's complexity and improving performance were two
of the main reasons for the change. The rebalance protocol on top of
Zookeeper was difficult to implement correctly, and I think a number of
Kafka clients never actually got it working. Removing it as a dependence
also makes a lot of sense with the introduction of security in 0.9.
Zookeeper is easier to protect since it only needs to be accessed by the
brokers and the clients are easier to audit since all requests go through
Kafka. The full rationale is documented here:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design.

-Jason

On Sat, Dec 5, 2015 at 1:07 AM, Li Tao <ahumblecoder@gmail.com> wrote:

> Hi guys,
>
> As we know, the new consumer of 0.9 kafka does not need to communicate with
> zookeeper.
> What is the motive / rational behind this disign? I have some
> guesses(correct me if i am wrong):
>
> 1. Reduce the complexity of consumer. It's hard to maintain the connections
> and states of kafka servers and zookeeper servers.
>
> 2. Improve performance. Too many consumers querying and writting to
> zookeeper will decrease the performance as a whole.
>
> Thanks.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message