kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Su She <suhsheka...@gmail.com>
Subject Re: java.nio.channels.ClosedChannelException...Firewall Issue?
Date Tue, 20 Jan 2015 06:58:54 GMT
Hi Jaikiran,

Thanks for the reply!

1) I started Kafka server on instance A by simply downloading
Kafka_2.10-0.8.2-beta.tgz from the kafka website, and using the scripts
mentioned here: http://kafka.apache.org/documentation.html#introduction.
This is the same way I downloaded Kafka on B, except I only started the
producer, not the ZK or Server.

2) How do I check to see if the services are bound to non localhost IP?

3) I believe it is firewall because I started 2 instances (C and D) using
the AWS Quick Start (
https://s3.amazonaws.com/quickstart-reference/cloudera/hadoop/latest/doc/Cloudera_EDH_on_AWS.pdf).
I was able to download Kafka on C and D and pass messages using Kafka
Producer/Consumer from D to C using the private/public ip address of C
(which was the instance I was running the consumer/ZK/Server on). Since I
used AWS Quick Start both of these were in a private subnet.

Edit: Woops, I saw I said B was an ec2-instance, B is a gateway, and I
downloaded Kafka same way I downloaded it on EC2 and I started a producer
on the gateway hoping to send it to my EC2 instance. I can ping A (in a
private subnet) from my gateway fine. I have pretty much changed all
security settings on A so that it essentially has none, but still in a
private subnet ( so can't ssh into it still from putty, can only ssh via
the cluster launcher). I can't lessen the security anymore on instance A.

Hope that wasn't too confusing, really appreciate the help. Thank you!

Best,

Su



On Mon, Jan 19, 2015 at 7:58 PM, Jaikiran Pai <jai.forums2013@gmail.com>
wrote:

> Hi Su,
>
> How exactly did you start the Kafka server on instance "A"? Are you sure
> the services on it are bound to non localhost IP? What does the following
> command result from instance B:
>
> telnet public.ip.of.A 9092
>
>
> -Jaikiran
>
> On Tuesday 20 January 2015 07:16 AM, Su She wrote:
>
>> Hello Everyone,
>>
>> Thank you for the help!
>>
>> Preface: I've created producers/consumers before and they have worked. I
>> have also made consumers/producers using java programs, but they have all
>> been locally.
>>
>> 1) I have a Zookeeper/Kafka Server running on an EC2 instance called "A"
>>
>> 2) I started the Zookeeper/Kafka Server on A and created a topic "test"
>> like it says on the kafka documentation
>>
>> 3) I then started a console consumer on A like the documentation states
>> for
>> test.
>>
>> 4) I then downloaded Kafka on EC2 instance called "B" and created a
>> console
>> producer with broker with broker-list public.ip.of.A:9092 and topic test.
>>
>> 5) Once I start publishing messages this is the message I get (what's
>> worse
>> is that I can't cntrl C to stop and have to exit putty):
>>
>> WARN Fetching topic metadata with correlation id 16 for topics [Set(test)]
>> from broker [id:0,host:54.183.40.224,port:9092] failed
>> (kafka.client.ClientUtils$)
>>
>> java.nio.channels.ClosedChannelException
>>
>>          at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
>>
>>          at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.
>> scala:74)
>>
>>          at
>> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$
>> doSend(SyncProducer.scala:73)
>>
>>          at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
>>
>>          at
>> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
>>
>>          at
>> kafka.producer.BrokerPartitionInfo.updateInfo(
>> BrokerPartitionInfo.scala:82)
>>
>>          at
>> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(
>> DefaultEventHandler.scala:78)
>>
>>          at kafka.utils.Utils$.swallow(Utils.scala:172)
>>
>>          at kafka.utils.Logging$class.swallowError(Logging.scala:106)
>>
>>          at kafka.utils.Utils$.swallowError(Utils.scala:45)
>>
>>          at
>> kafka.producer.async.DefaultEventHandler.handle(
>> DefaultEventHandler.scala:78)
>>
>>          at
>> kafka.producer.async.ProducerSendThread.tryToHandle(
>> ProducerSendThread.scala:104)
>>
>>          at
>> kafka.producer.async.ProducerSendThread.processEvents(
>> ProducerSendThread.scala:93)
>>
>>          at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>>
>> [2015-01-20 01:29:28,289] ERROR fetching topic metadata for topics
>> [Set(test)] from broker [ArrayBuffer(id:0,host:
>>
>> Meanwhile in ZK...not sure if this happened as soon as I started
>> publishing
>> messages:
>>
>> Error:KeeperErrorCode = NoNode for /consumers/console-consumer-
>> 2615/offsets
>> (org.apache.zookeeper.server.PrepRequestProcessor)
>>
>> Any tips/suggestions are greatly appreciated!
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message