kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From A A <andthereitg...@hotmail.com>
Subject RE: Remote Zookeeper
Date Tue, 11 Mar 2014 20:25:55 GMT
Yes. True. Thanks for your help.

> Date: Tue, 11 Mar 2014 11:48:20 -0700
> Subject: Re: Remote Zookeeper
> From: b@b3k.us
> To: users@kafka.apache.org
> 
> If they are on different physical machines then binding to localhost/using
> localhost as the host name is unlikely to be what you want.
> 
> On Tuesday, March 11, 2014, A A <andthereitgoes@hotmail.com> wrote:
> 
> > Thanks. I already checked out the wiki and step 6 in particular.
> > Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical
> > machines.
> >
> > > From: b@b3k.us <javascript:;>
> > > Date: Tue, 11 Mar 2014 10:58:50 -0700
> > > Subject: Re: Remote Zookeeper
> > > To: users@kafka.apache.org <javascript:;>
> > >
> > > "I'd suggest deleting the zookeeper and Kafka logs and starting over
> > using
> > > the getting started tutorial from the wiki."
> > >
> > >    |
> > >    v
> > >
> > > "Could this be an issue?
> > > INFO Registered broker 2 at path /brokers/ids/2 with address
> > localhost:9092.
> > > INFO Registered broker 1 at path /brokers/ids/1 with address
> > > localhost:9092."
> > >
> > > yes. see step 6: https://kafka.apache.org/documentation.html#quickstart
> > >
> > >
> > >
> > > On Tue, Mar 11, 2014 at 10:54 AM, A A <andthereitgoes@hotmail.com<javascript:;>>
> > wrote:
> > >
> > > > Okay thanks. I removed zookeeper and kafka logs
> > > >
> > > > /tmp/kafka-logs
> > > > /tmp/logs
> > > > /var/zookeeper/version-2/
> > > > /tmp/zookeeper*.log (on Zk)
> > > >
> > > > restarted zookeeper
> > > >
> > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > > no topics exist!
> > > >
> > > > Started Broker 2
> > > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > > [2014-03-11 12:05:32,651] INFO Verifying properties
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is
> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to
> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is
> > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is
> > > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is
> > overridden to
> > > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is
> > overridden
> > > > to 168 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property num.network.threads is
> > overridden
> > > > to 2 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property
> > zookeeper.connection.timeout.msis overridden to 1000000
> > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is
> > > > overridden to 10000 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is
> > overridden to
> > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting
> > > > (kafka.server.KafkaServer)
> > > > [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log
> > > > cleaner every 60000 ms (kafka.log.LogManager)
> > > > [2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log
> > > > flusher every 3000 ms with the following overrides Map()
> > > > (kafka.log.LogManager)
> > > > [2014-03-11 12:05:32,791] INFO Awaiting socket connections on
> > 0.0.0.0:9092.
> > > > (kafka.network.Acceptor)
> > > > [2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started
> > > > (kafka.network.SocketServer)
> > > > [2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:2181
> > (kafka.server.KafkaZooKeeper)
> > > > [2014-03-11 12:05:32,815] INFO Starting ZkClient event thread.
> > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > [2014-03-11 12:05:32,817] INFO Client
> > > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47
> > GMT
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:java.vendor=Oracle
> > > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client
> > > >
> > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client
> > > >
> > environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client
> > > >
> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:java.io.tmpdir=/tmp
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:java.compiler=<NA>
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:os.name=Linux
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:os.arch=amd64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client
> > > > environment:os.version=2.6.32-279.el6.x86_64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:user.name=root
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:user.home=/root
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:user.dir=/opt
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,825] INFO Initiating client connection,
> > connectString=
> > > > 192.168.1.120:2181 sessionTimeout=6000
> > > > watcher=org.I0Itec.zkclient.ZkClient@721e34d8
> > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,842] INFO Opening socket connection to server /
> > > > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:05:32,846] INFO Socket connection established to
> > > > 192.168.1.120/192.168.1.120:2181, initiating session
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:05:32,854] INFO Session establishment complete on server
> > > > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210004,
> > > > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:05:32,856] INFO zookeeper state changed (SyncConnected)
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2014-03-11 12:05:32,891] INFO Registered broker 2 at path
> > /brokers/ids/2
> > > > with address localhost:9092. (kafka.utils.ZkUtils$)
> > > > [2014-03-11 12:05:32,892] INFO [Kafka Server 2], Connecting to ZK:
> > > > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > > > [2014-03-11 12:05:33,031] INFO Will not load MX4J, mx4j-tools.jar is
> > not
> > > > in the classpath (kafka.utils.Mx4jLoader$)
> > > > [2014-03-11 12:05:33,041] INFO 2 successfully elected as leader
> > > > (kafka.server.ZookeeperLeaderElector)
> > > > [2014-03-11 12:05:33,371] INFO [Kafka Server 2], Started
> > > > (kafka.server.KafkaServer)
> > > > [2014-03-11 12:05:33,377] INFO New leader is 2
> > > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > > >
> > > > Then started Broker 1
> > > >
> > > >  $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > > [2014-03-11 12:07:20,843] INFO Verifying properties
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,875] INFO Property port is overridden to 9092
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,876] INFO Property socket.request.max.bytes is
> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,876] INFO Property num.io.threads is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,876] INFO Property log.dirs is overridden to
> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,876] INFO Property log.cleanup.interval.mins is
> > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property socket.send.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property log.flush.interval.ms is
> > > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property zookeeper.connect is
> > overridden to
> > > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property broker.id is overridden to 1
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property log.retention.hours is
> > overridden
> > > > to 168 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,878] INFO Property num.network.threads is
> > overridden
> > > > to 2 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,878] INFO Property socket.receive.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,878] INFO Property
> > zookeeper.connection.timeout.msis overridden to 1000000
> > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,878] INFO Property num.partitions is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,879] INFO Property log.flush.interval.messages is
> > > > overridden to 10000 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,879] INFO Property log.segment.bytes is
> > overridden to
> > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,897] INFO [Kafka Server 1], Starting
> > > > (kafka.server.KafkaServer)
> > > > [2014-03-11 12:07:20,922] INFO [Log Manager on Broker 1] Log directory
> > > > '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
> > > > [2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log
> > > > cleaner every 60000 ms (kafka.log.LogManager)
> > > > [2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log
> > > > flusher every 3000 ms with the following overrides Map()
> > > > (kafka.log.LogManager)
> > > > [2014-03-11 12:07:20,991] INFO Awaiting socket connections on
> > 0.0.0.0:9092.
> > > > (kafka.network.Acceptor)
> > > > [2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started
> > > > (kafka.network.SocketServer)
> > > > [2014-03-11 12:07:21,002] INFO connecting to ZK: 192.168.1.120:2181
> > (kafka.server.KafkaZooKeeper)
> > > > [2014-03-11 12:07:21,018] INFO Starting ZkClient event thread.
> > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > [2014-03-11 12:07:21,029] INFO Client
> > > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47
> > GMT
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle
> > > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client
> > > >
> > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client
> > > >
> > environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client
> > > >
> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:java.compiler=<NA>
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client
> > > > environment:os.version=2.6.32-279.el6.x86_64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:user.name=root
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client
> > environment:user.dir=/var/zookeeper
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,031] INFO Initiating client connection,
> > connectString=
> > > > 192.168.1.120:2181 sessionTimeout=6000
> > > > watcher=org.I0Itec.zkclient.ZkClient@45419cee
> > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,043] INFO Opening socket connection to server /
> > > > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:07:21,049] INFO Socket connection established to
> > > > 192.168.1.120/192.168.1.120:2181, initiating session
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:07:21,058] INFO Session establishment complete on server
> > > > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005,
> > > > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2014-03-11 12:07:21,095] INFO Registered broker 1 at path
> > /brokers/ids/1
> > > > with address localhost:9092. (kafka.utils.ZkUtils$)
> > > > [2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
> > > > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > > > [2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is
> > not
> > > > in the classpath (kafka.utils.Mx4jLoader$)
> > > > [2014-03-11 12:07:21,249] INFO conflict in /controller data: {
> > > > "brokerid":1, "timestamp":"1394539641229", "version":1 } stored data:
{
> > > > "brokerid":2, "timestamp":"1394539533037", "version":1 }
> > > > (kafka.utils.ZkUtils$)
> > > > [2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
> > > > (kafka.server.KafkaServer)
> > > >
> > > > Could this be an issue?
> > > > INFO Registered broker 2 at path /brokers/ids/2 with address
> > > > localhost:9092.
> > > > INFO Registered broker 1 at path /brokers/ids/1 wi
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message