kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: multi-node and multi-broker kafka cluster setup
Date Tue, 30 Sep 2014 15:35:03 GMT

In general it is not required to have the kafka brokers installed on the
same nodes of the zk servers, and each node can host multiple kafka
brokers: you just need to make sure they do not share the same port and the
same data dir.


On Mon, Sep 29, 2014 at 8:31 PM, Sa Li <salicn@gmail.com> wrote:

> Hi,
> I am kinda newbie to kafka, I plan to build a cluster with multiple nodes,
> and multiple brokers on each node, I can find tutorials for set multiple
> brokers cluster in single node, say
> http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/
> Also I can find some instructions for multiple node setup, but with single
> broker on each node. I have not seen any documents to teach me how to setup
> multiple nodes cluster and multiple brokers in each node. I notice some
> documents points out: we should install kafka on each node which makes
> sense, and all the brokers in each node should connect to same zookeeper. I
> am confused since I thought I could setup a zookeeper ensemble cluster
> separately, and all the brokers connecting to this zookeeper cluster and
> this zk cluster doesn’t have to be the server hosting the kafka, but some
> tutorial says I should install zookeeper on each kafka node.
> Here is my plan:
> - I have three nodes: kfServer1, kfserver2, kfserver3,
> - kfserver1 and kfserver2 are configured as the zookeeper ensemble, which
> i have done.
>   zk.connect=kfserver1:2181,kfserver2:2181
> - broker1, broker2, broker3 are in kfserver1,
>   broker4, broker5, broker6 are on kfserver2,
>   broker7, broker8, broker9 are on kfserver3.
> When I am configuring, the zk DataDir is in local directory of each node,
> instead located at the zk ensemble directory, is that correct? So far, I
> couldnot make above scheme working, anyone have ever made multi-node and
> multi-broker kafka cluster setup?
> thanks
> Alec

-- Guozhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message