kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: How produce to remote Kafka Cluster over ssh tunneling
Date Wed, 09 Apr 2014 04:03:14 GMT
Any error in the broker, controller and state change log?

Thanks,

Jun


On Tue, Apr 8, 2014 at 3:08 PM, Sotaro Kimura <rfbringer@gmail.com> wrote:

> Hi, all.
>
> Sorry for my late reply.
>
> I tried advertised.host.name/advertised.port.
> But producer failed fetching metadata from kafka broker.
>
> Configuration is below.
> - Kafka cluster in DataCenter(each machine has Grobal IP Address).
> Firewall allows only ssh port(22).
> ex) 192.168.100.100
> - Kafka producer exists out of DataCenter.
> ex) 172.16.0.100
>
> Broker additional setting is below.
> advertised.host.name=localhost
> advertised.port=19092
>
> Kafka producer operation is below.
>
> - SSH port fowarding
> ========================
> # ssh root@192.168.100.100 -L 19092:192.168.100.100:9092 -L 12181:
> 192.168.100.100:2181
> // Kafka broker and zookeeper ssh port fowarding.
> ========================
>
> - Producer data put(failed)
> ========================
> # ./kafka-topics.sh --create --zookeeper localhost:12181
> --replication-factor 1 --partitions 3 --topic TestTopic
> Created topic "TestTopic".
> // Succeed create topic over ssh port fowarding.
> # ./kafka-topics.sh --list --zookeeper localhost:12181
> TestTopic
> // Succeed get topic list over ssh port fowarding.
> # ./kafka-console-producer.sh --broker-list localhost:19092 --topic
> TestTopic
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Test
> [2014-04-06 20:42:01,352] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-04-06 20:42:01,364] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-04-06 20:42:01,365] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: TestTopic
> (kafka.producer.async.DefaultEventHandler)
> [2014-04-06 20:42:01,476] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-04-06 20:42:01,481] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-04-06 20:42:01,481] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: TestTopic
> (kafka.producer.async.DefaultEventHandler)
> [2014-04-06 20:42:01,589] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-04-06 20:42:01,595] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-04-06 20:42:01,595] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: TestTopic
> (kafka.producer.async.DefaultEventHandler)
> [2014-04-06 20:42:01,709] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartit
> ionInfo)
> [2014-04-06 20:42:01,719] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-04-06 20:42:01,719] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: TestTopic
> (kafka.producer.async.DefaultEventHandler)
> [2014-04-06 20:42:01,826] WARN Error while fetching metadata
> [{TopicMetadata for topic TestTopic ->No partition metadata for topic
> TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
> [TestTopic]: class kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-04-06 20:42:01,829] ERROR Failed to send requests for topics
> TestTopic with correlation ids in [0,8]
> (kafka.producer.async.DefaultEventHandler)
> [2014-04-06 20:42:01,830] ERROR Error in handling batch of 2 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:526)
> at
>
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> ========================
>
>
> If advertised setting is not defined,
> producer connection is refused.(below error)
> So I think advertised setting is working.
>
> ========================
> # ./kafka-console-producer.sh --broker-list localhost:19092 --topic
> TestTopic
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Test
> [2014-04-08 21:11:44,231] ERROR Producer connection to localhost:9092
> unsuccessful (kafka.producer.SyncProducer)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> ========================
>
> Is there anyone you find the cause of error.
>
> regards.
>
>
>
>
> 2014-03-20 8:00 GMT+09:00 Sotaro Kimura <rfbringer@gmail.com>:
>
> > Thanks for quick reply.
> >
> >
> > advertised.host.name/advertised.port setting is this.
> > I confirmed only producer setting, so I couldn't find these properties.
> >
> > I will try it. Thank you.
> >
> >
> > 2014-03-20 7:50 GMT+09:00 Joe Stein <joe.stein@stealth.ly>:
> >
> > With 0.8.1 update these two properties in the server.properties config of
> >> the broker to be 172.16.0.100 and 19092 respectively
> >>
> >> # Hostname the broker will advertise to producers and consumers. If not
> >> set, it uses the
> >> # value for "host.name" if configured.  Otherwise, it will use the
> value
> >> returned from
> >> # java.net.InetAddress.getCanonicalHostName().
> >> #advertised.host.name=<hostname routable by clients>
> >>
> >> # The port to publish to ZooKeeper for clients to use. If this is not
> set,
> >> # it will publish the same port that the broker binds to.
> >> #advertised.port=<port accessible by clients>
> >>
> >> /*******************************************
> >>  Joe Stein
> >>  Founder, Principal Consultant
> >>  Big Data Open Source Security LLC
> >>  http://www.stealth.ly
> >>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> >> ********************************************/
> >>
> >>
> >> On Wed, Mar 19, 2014 at 6:39 PM, Sotaro Kimura <rfbringer@gmail.com>
> >> wrote:
> >>
> >> > Hi, all.
> >> >
> >> > I'm trying Log produce to Remote Kafka Cluster over SSH Tunneling.
> >> >
> >> > Configuration is below.
> >> > - Kafka cluster in DataCenter(each machine has Grobal IP Address).
> >> > Firewall allows only ssh port(22).
> >> > ex) 192.168.100.100
> >> > - Kafka producer exists out of DataCenter.
> >> > ex) 172.16.0.100
> >> >
> >> > I tried produce over SSH Tunneling.
> >> > Producer successed getting meta data from brokers.
> >> > But meta data is Address which producer can't access directry.
> >> >
> >> > Detail is below.
> >> > 1. Ssh connect producer machine to DataCenter.
> >> > 172.16.0.100 > 192.168.100.100:22
> >> > 2. Create SSH Tunnel over ssh connection.
> >> > 172.16.0.100's localhost:19092 to 192.168.100.100:9092(Kafka receive
> >> > port).
> >> > 3. Producer connect to localhost:19092.
> >> > Producer get metadata from brokers(192.168.100.100:9092).
> >> > 4. Producer failed to connect brokers(192.168.100.100:9092).
> >> >
> >> > In such situations, how I can produce to Remote Kafka Cluster?
> >> >
> >> > --
> >> > #################################
> >> > 木村 宗太郎 Sotaro Kimura
> >> > <rfbringer@gmail.com>
> >> > #################################
> >> >
> >>
> >
> >
> >
> > --
> > #################################
> > 木村 宗太郎 Sotaro Kimura
> > <rfbringer@gmail.com>
> > #################################
> >
>
>
>
> --
> #################################
> 木村 宗太郎 Sotaro Kimura
> <rfbringer@gmail.com>
> #################################
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message