spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <luohui20...@sina.com>
Subject SparkStreaming + Flume: org.jboss.netty.channel.ChannelException: Failed to bind to: master60/10.0.10.60:31001
Date Thu, 25 Aug 2016 15:21:06 GMT
Hi there      I have a flume cluster sending messages to SparkStreaming. I got an exception
like below:16/08/25 23:00:54 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error
starting receiver 0 - org.jboss.netty.channel.ChannelException: Failed to bind to: master60/10.0.10.60:31001
        at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:106)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:119)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:74)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:68)
        at org.apache.spark.streaming.flume.FlumeReceiver.initServer(FlumeInputDStream.scala:162)
        at org.apache.spark.streaming.flume.FlumeReceiver.onStart(FlumeInputDStream.scala:169)
        at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
        at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:130)
        at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:575)
        at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:565)
        at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
        at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
        at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
        ... 3 more


I checked the port like below:[hadoop@master60 shellscripts]$ netstat -an|grep 31001
tcp        0      0 :::31001                    :::*                        LISTEN
tcp        0      0 ::ffff:10.0.10.60:31001     ::ffff:10.0.30.199:34773    ESTABLISHED

It seems that the workers on other nodes(except the one on master60) could not connect the
port 31001 on master60 However I tried[hadoop@slave61 ~]$ telnet master60 31001
Trying 10.0.10.60...
Connected to master60.
Escape character is '^]'.
So I didn't quit get why the executors can not build connections?BTW, I am using Spark1.6.1,
flume 1.6.0
Any idea will be appreciated.



--------------------------------

 

Thanks&amp;Best regards!
San.Luo
Mime
View raw message