Yeah, seems to be the case. In general your executors should be able to reach the driver, which I don't think is the case for you currently (LinuxDevVM.local:59266 seems very private). What you need is some sort of gateway node that can be publicly reached from your worker machines to launch your driver.

Andrew


2014-07-23 10:40 GMT-07:00 didi <didist@gmail.com>:
Hi all
I guess the problem has something to do with the fact i submit the job to
remote location

I submit from OracleVM running ubuntu and suspect some NAT issues maybe?
akka tcp tries this address as follows from the STDERR print which is
appended akka.tcp://spark@LinuxDevVM.local:59266

STDERR PRINT:
--------------

Spark Executor Command: "java" "-cp"
"::/root/Downloads/spark-1.0.1-bin-hadoop2/conf:/root/Downloads/spark-1.0.1-bin-hadoop2/lib/spark-assembly-1.0.1-hadoop2.2.0.jar"
"-XX:MaxPermSize=128m" "-Xms1024M" "-Xmx1024M"
"org.apache.spark.executor.CoarseGrainedExecutorBackend"
"akka.tcp://spark@LinuxDevVM.local:59266/user/CoarseGrainedScheduler" "3"
"bigdata-1.comp.com" "4"
"akka.tcp://sparkWorker@bigdata-1.comp.com:52497/user/Worker"
"app-20140723132701-0012"
========================================

log4j:WARN No appenders could be found for logger
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
14/07/23 13:27:04 INFO SparkHadoopUtil: Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
14/07/23 13:27:04 INFO SecurityManager: Changing view acls to: root
14/07/23 13:27:04 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root)
14/07/23 13:27:05 INFO Slf4jLogger: Slf4jLogger started
14/07/23 13:27:05 INFO Remoting: Starting remoting
14/07/23 13:27:05 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkExecutor@bigdata-1.comp.com:53886]
14/07/23 13:27:05 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sparkExecutor@il-bigdata-1.comp.com:53886]
14/07/23 13:27:05 INFO CoarseGrainedExecutorBackend: Connecting to driver:
akka.tcp://spark@LinuxDevVM.local:59266/user/CoarseGrainedScheduler
14/07/23 13:27:05 INFO WorkerWatcher: Connecting to worker
akka.tcp://sparkWorker@bigdata-1.comp.com:52497/user/Worker
14/07/23 13:27:05 WARN Remoting: Tried to associate with unreachable remote
address [akka.tcp://spark@LinuxDevVM.local:59266]. Address is now gated for
60000 ms, all messages to this address will be delivered to dead letters.
14/07/23 13:27:05 ERROR CoarseGrainedExecutorBackend: Driver Disassociated
[akka.tcp://sparkExecutor@bigdata-1.comp.com:53886] ->
[akka.tcp://spark@LinuxDevVM.local:59266] disassociated! Shutting down.
-------------------------------------
Thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-to-remote-master-fails-tp10522.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.