kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From A A <andthereitg...@hotmail.com>
Subject RE: Remote Zookeeper
Date Tue, 11 Mar 2014 16:45:34 GMT
I noticed a problem with my topics. The previous two topics would only be present on 1 broker.
 So I created a new topic

$KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181 --replica 2 --partition
1 --topic test-rep
$KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
topic: test     partition: 0    leader: 1       replicas: 1     isr: 1
topic: test-rep partition: 0    leader: 1       replicas: 1,2   isr: 1
topic: test1    partition: 0    leader: 1       replicas: 1     isr: 1

No there is no firewall. Even then I stopped the iptables.  And then I tried the following

 $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.123:9092,192.168.1.124:9092
--topic test-rep
replicated message

[2014-03-11 10:59:25,844] WARN Error while fetching metadata [{TopicMetadata for topic test-rep
->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}]
for topic [test-rep]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,867] WARN Error while fetching metadata [{TopicMetadata for topic test-rep
->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}]
for topic [test-rep]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic, partition due to: Failed
to fetch topic metadata for topic: test-rep (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:59:25,979] WARN Error while fetching metadata [{TopicMetadata for topic test-rep
->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}]
for topic [test-rep]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,986] WARN Error while fetching metadata [{TopicMetadata for topic test-rep
->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}]
for topic [test-rep]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic, partition due to: Failed
to fetch topic metadata for topic: test-rep (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:59:26,098] WARN Error while fetching metadata [{TopicMetadata for topic test-rep
->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}]
for topic [test-rep]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)




> Date: Tue, 11 Mar 2014 09:05:54 -0700
> Subject: Re: Remote Zookeeper
> From: pradeepg26@gmail.com
> To: users@kafka.apache.org
> 
> Is there a firewall thats blocking connections on port 9092? Also, the
> broker list should be comma separated.
> 
> 
> On Tue, Mar 11, 2014 at 9:02 AM, A A <andthereitgoes@hotmail.com> wrote:
> 
> > Sorry one of the brokers for was down. Brought it back up. Tried the
> > following
> >
> >  $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > 192.168.1.124:9092 --topic test
> > hello brokers
> >
> > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:56,059] ERROR Failed to send requests for topics test
> > with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:56,061] ERROR Error in handling batch of 1 events
> > (kafka.producer.async.ProducerSendThread)
> > kafka.common.FailedToSendMessageException: Failed to send messages after 3
> > tries.
> >         at
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> >         at
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> >         at
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> >         at
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> >         at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> >         at
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> >         at
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> >
> >
> >
> > From: andthereitgoes@hotmail.com
> > To: users@kafka.apache.org
> > Subject: RE: Remote Zookeeper
> > Date: Tue, 11 Mar 2014 15:59:44 +0000
> >
> >
> >
> >
> > Okay thanks. Just to verify my setup I tried the following on broker1 (by
> > publishing to localhost)
> >
> >  $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > --topic test
> > test message
> > this is a test message
> > another test message
> >
> > $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.1.120:2181--topic
test --from-beginning
> > test message
> > this is a test message
> > another test message
> > Consumed 3 messages
> >
> > This works fine. How do I provide the broker-list?
> > I tried the following on broker1 (192.168.1.123)
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092
> > 192.168.1.123:9092 --topic test
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092
> > ,192.168.1.123:9092 --topic test
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > 192.168.1.123:9092 --topic test
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092,
> > 192.168.1.123:9092 --topic test
> >
> > Didn't work. I either get connection refused or no route to host
> > exceptions.
> >
> > > Date: Tue, 11 Mar 2014 08:38:22 -0700
> > > Subject: Re: Remote Zookeeper
> > > From: b@b3k.us
> > > To: users@kafka.apache.org
> > >
> > > That's not how Kafka works. You need to pass the full list of brokers.
> > >
> > > On Tuesday, March 11, 2014, A A <andthereitgoes@hotmail.com> wrote:
> > >
> > > > Hi again.
> > > >
> > > > Got the setup working. I now have 2 brokers (broker 1 and broker 2)
> > with
> > > > one remote zk. I was also able to create some topics
> > > >
> > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > > topic: test     partition: 0    leader: 1       replicas: 1     isr: 1
> > > > topic: test1    partition: 0    leader: 1       replicas: 1     isr: 1
> > > >
> > > > I then try to publish a message from broker1 to broker 1 (localhost)
> > while
> > > > I am consuming on broker2
> > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > > > --topic test1
> > > > helloooo
> > > > [2014-03-11 07:36:35,206] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,244] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,248] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test1
> > > > (kafka.producer.async.Default
> > > >  EventHandler)
> > > > [2014-03-11 07:36:35,358] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,368] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,368] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test1
> > > > (kafka.producer.async.Default
> > > >  EventHandler)
> > > > [2014-03-11 07:36:35,476] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,490] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,490] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test1
> > > > (kafka.producer.async.Default
> > > >  EventHandler)
> > > > [2014-03-11 07:36:35,599] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,609] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,609] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test1
> > > > (kafka.producer.async.Default
> > > >  EventHandler)
> > > > [2014-03-11 07:36:35,721] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException  (kafka.pr
> > > >                oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,723] ERROR Failed to send requests for topics
> > test1
> > > > with correlation ids in [0,8]
> > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 07:36:35,725] ERROR Error in handling batch of 1 events
> > > > (kafka.producer.async.ProducerSendThread)
> > > > kafka.common.FailedToSendMessageException: Failed to send messages
> > after 3
> > > > tries.
> > > >         at
> > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > >         at
> > > >
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > >         at
> > > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > >         at
> > > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > >         at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > > >         at
> > > >
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > >         at
> > > >
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > >
> > > > What am I missing?
> > > >
> > > > > From: andthereitgoes@hotmail.com <javascript:;>
> > > > > To: users@kafka.apache.org <javascript:;>
> > > > > Subject: RE: Remote Zookeeper
> > > > > Date: Tue, 11 Mar 2014 02:29:34 +0000
> > > > >
> > > > > Thanks. Totally missed that.
> > > > >
> > > > > > From: b@b3k.us <javascript:;>
> > > > > > Date: Mon, 10 Mar 2014 19:18:50 -0700
> > > > > > Subject: Re: Remote Zookeeper
> > > > > > To: users@kafka.apache.org <javascript:;>
> > > > > >
> > > > > > zookeeper.connect
> > > > > >
> > > > > > https://kafka.apache.org/08/configuration.html
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 10, 2014 at 7:17 PM, A A <andthereitgoes@hotmail.com
> > <javascript:;>>
> > > > wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > Pretty new to Kafka. Have been successful in installing
Kafka
> > 0.8.0.
> > > > > > > I am just wondering how should I make my kafka cluster
(2
> > brokers)
> > > > connect
> > > > > > > to a single remote zookeper server?
> > > > > > >
> > > > > > > I am using  $KAFKA/kafka-server-start.sh
> > > > $KAFKA_CONFIG/server.properties
> > > > > > > on both the brokers to start them up
> > > > > > > and I have a remote zookeeper running at 192.168.1.120
> > > > > > >
> > > > > > > Can anyone help?
> > > > > > >
> > > > > > > A
> > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> >
> >
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message