spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi.Zhang" <>
Subject Why association with remote system has failed when set master in Spark programmatically
Date Sat, 16 May 2015 04:00:42 GMT
Hi all,

I run to start standalone Spark with
spark:// Then, I use this command as below, and it's OK:
./bin/spark-shell --master spark://

The console print correct message, and Spark context had been initialised

However, when I run app in IntelliJ Idea using spark conf like this:
val sparkConf = new SparkConf().setAppName("FromMySql")
      .set("spark.akka.heartbeat.interval", "100")
    val sc = new SparkContext(sparkConf)
    val sqlContext = new SQLContext(sc) 

It can't talk to spark and print these error messages:
ReliableDeliverySupervisor: Association with remote system
[akka.tcp://sparkMaster@] has failed, address is now gated
for [5000] ms. Reason is: [Disassociated].

If I changed the conf to local[*], it's ok. After I packaged my app then use
spark-submit command, the communication between local and remote actor is
OK.  It's very strange! 

Then I debugged it, and the remote actor can be fetched correctly in  the
tryRegisterAllMasters() method of AppClient:
    def tryRegisterAllMasters() {
      for (masterAkkaUrl <- masterAkkaUrls) {
        logInfo("Connecting to master " + masterAkkaUrl + "...")
        val actor = context.actorSelection(masterAkkaUrl)
        actor ! RegisterApplication(appDescription)

After actor send the RegisterApplication message, it seems like the message
is not routed to the remote actor, so registering operation is not finished,
then failed. 

I don't know what is the reason. Who know the answer?


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message