kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sotaro Kimura <rfbrin...@gmail.com>
Subject Re: How produce to remote Kafka Cluster over ssh tunneling
Date Tue, 08 Apr 2014 22:08:55 GMT
Hi, all.

Sorry for my late reply.

I tried advertised.host.name/advertised.port.
But producer failed fetching metadata from kafka broker.

Configuration is below.
- Kafka cluster in DataCenter(each machine has Grobal IP Address).
Firewall allows only ssh port(22).
ex) 192.168.100.100
- Kafka producer exists out of DataCenter.
ex) 172.16.0.100

Broker additional setting is below.
advertised.host.name=localhost
advertised.port=19092

Kafka producer operation is below.

- SSH port fowarding
========================
# ssh root@192.168.100.100 -L 19092:192.168.100.100:9092 -L 12181:
192.168.100.100:2181
// Kafka broker and zookeeper ssh port fowarding.
========================

- Producer data put(failed)
========================
# ./kafka-topics.sh --create --zookeeper localhost:12181
--replication-factor 1 --partitions 3 --topic TestTopic
Created topic "TestTopic".
// Succeed create topic over ssh port fowarding.
# ./kafka-topics.sh --list --zookeeper localhost:12181
TestTopic
// Succeed get topic list over ssh port fowarding.
# ./kafka-console-producer.sh --broker-list localhost:19092 --topic
TestTopic
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Test
[2014-04-06 20:42:01,352] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-04-06 20:42:01,364] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-04-06 20:42:01,365] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: TestTopic
(kafka.producer.async.DefaultEventHandler)
[2014-04-06 20:42:01,476] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-04-06 20:42:01,481] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-04-06 20:42:01,481] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: TestTopic
(kafka.producer.async.DefaultEventHandler)
[2014-04-06 20:42:01,589] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-04-06 20:42:01,595] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-04-06 20:42:01,595] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: TestTopic
(kafka.producer.async.DefaultEventHandler)
[2014-04-06 20:42:01,709] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartit
ionInfo)
[2014-04-06 20:42:01,719] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-04-06 20:42:01,719] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: TestTopic
(kafka.producer.async.DefaultEventHandler)
[2014-04-06 20:42:01,826] WARN Error while fetching metadata
[{TopicMetadata for topic TestTopic ->No partition metadata for topic
TestTopic due to kafka.common.LeaderNotAvailableException}] for topic
[TestTopic]: class kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-04-06 20:42:01,829] ERROR Failed to send requests for topics
TestTopic with correlation ids in [0,8]
(kafka.producer.async.DefaultEventHandler)
[2014-04-06 20:42:01,830] ERROR Error in handling batch of 2 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
========================


If advertised setting is not defined,
producer connection is refused.(below error)
So I think advertised setting is working.

========================
# ./kafka-console-producer.sh --broker-list localhost:19092 --topic
TestTopic
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Test
[2014-04-08 21:11:44,231] ERROR Producer connection to localhost:9092
unsuccessful (kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
========================

Is there anyone you find the cause of error.

regards.




2014-03-20 8:00 GMT+09:00 Sotaro Kimura <rfbringer@gmail.com>:

> Thanks for quick reply.
>
>
> advertised.host.name/advertised.port setting is this.
> I confirmed only producer setting, so I couldn't find these properties.
>
> I will try it. Thank you.
>
>
> 2014-03-20 7:50 GMT+09:00 Joe Stein <joe.stein@stealth.ly>:
>
> With 0.8.1 update these two properties in the server.properties config of
>> the broker to be 172.16.0.100 and 19092 respectively
>>
>> # Hostname the broker will advertise to producers and consumers. If not
>> set, it uses the
>> # value for "host.name" if configured.  Otherwise, it will use the value
>> returned from
>> # java.net.InetAddress.getCanonicalHostName().
>> #advertised.host.name=<hostname routable by clients>
>>
>> # The port to publish to ZooKeeper for clients to use. If this is not set,
>> # it will publish the same port that the broker binds to.
>> #advertised.port=<port accessible by clients>
>>
>> /*******************************************
>>  Joe Stein
>>  Founder, Principal Consultant
>>  Big Data Open Source Security LLC
>>  http://www.stealth.ly
>>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>> ********************************************/
>>
>>
>> On Wed, Mar 19, 2014 at 6:39 PM, Sotaro Kimura <rfbringer@gmail.com>
>> wrote:
>>
>> > Hi, all.
>> >
>> > I'm trying Log produce to Remote Kafka Cluster over SSH Tunneling.
>> >
>> > Configuration is below.
>> > - Kafka cluster in DataCenter(each machine has Grobal IP Address).
>> > Firewall allows only ssh port(22).
>> > ex) 192.168.100.100
>> > - Kafka producer exists out of DataCenter.
>> > ex) 172.16.0.100
>> >
>> > I tried produce over SSH Tunneling.
>> > Producer successed getting meta data from brokers.
>> > But meta data is Address which producer can't access directry.
>> >
>> > Detail is below.
>> > 1. Ssh connect producer machine to DataCenter.
>> > 172.16.0.100 > 192.168.100.100:22
>> > 2. Create SSH Tunnel over ssh connection.
>> > 172.16.0.100's localhost:19092 to 192.168.100.100:9092(Kafka receive
>> > port).
>> > 3. Producer connect to localhost:19092.
>> > Producer get metadata from brokers(192.168.100.100:9092).
>> > 4. Producer failed to connect brokers(192.168.100.100:9092).
>> >
>> > In such situations, how I can produce to Remote Kafka Cluster?
>> >
>> > --
>> > #################################
>> > 木村 宗太郎 Sotaro Kimura
>> > <rfbringer@gmail.com>
>> > #################################
>> >
>>
>
>
>
> --
> #################################
> 木村 宗太郎 Sotaro Kimura
> <rfbringer@gmail.com>
> #################################
>



-- 
#################################
木村 宗太郎 Sotaro Kimura
<rfbringer@gmail.com>
#################################

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message