kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Swift <craig.sw...@returnpath.com.INVALID>
Subject Re: Running Zookeeper on same instance as kafka brokers in multi node cluster
Date Mon, 19 Sep 2016 21:48:30 GMT
Hello,

Initially we had our setup with both ZK and Kafka on the same nodes as
well. Over time though this proved problematic as we both increased the
usage of Kafka and had to scale out the cluster. ZK specifically can be
very touchy if it gets IO bound - so when the Kafka cluster would get under
high load we would start to see slow response and in the worst case
scenario a cascading failure. Also, from my understanding ZK doesn't really
scale well past five nodes - so once your cluster grows large enough your
nodes become non-homogeneous as you're only running ZK on certain nodes. So
from our experience it was much cleaner to spin up a 3 node ZK cluster for
each Kafka cluster we ran, dedicated to that cluster. That allows us to
grow out the Kafka cluster when needed without ever worrying about ZK. In
most cases we're more concerned about the ZK integrity and keeping that
solid and find it fairly easy to reprovision/add Kafka nodes when
necessary. Hope that helps.

Craig


On Mon, Sep 19, 2016 at 3:01 PM, Digumarthi, Prabhakar Venkata Surya <
PrabhakarVenkataSurya.Digumarthi@capitalone.com> wrote:

> Hi Team,
>
>
> What are the downsides of installing Zookeeper and kafka on same machine,
> in multi broker environment?
>
> We are trying to install Zookeeper and kafka in AWS world and its becoming
> difficult for us to maintain ZK and Kafka with some issues. Also
> re-provisioning  ZK and Kafka instances separately is getting complicated.
> Thanks,
> Prabhakar
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message