spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Jingci <cjcro...@gmail.com>
Subject RE: Spark with HDFS: ERROR Worker: Connection to master failed! Shuttingdown.
Date Sat, 02 Nov 2013 01:00:46 GMT
Can you try to use the IP address instead of the name 'base'? I also experience the same problem
before, it worked after changed to IP.

Thanks,
Chen jingci

-----Original Message-----
From: "Thorsten Bergler" <thorsten.bergler@tbonline.de>
Sent: ‎2/‎11/‎2013 2:03
To: "user@spark.incubator.apache.org" <user@spark.incubator.apache.org>
Subject: Spark with HDFS: ERROR Worker: Connection to master failed! Shuttingdown.

Hello,


I am new to Spark and doing my first steps with it today.

Right now I am having trouble with the error:

ERROR Worker: Connection to master failed! Shutting down.

So far, I found out the following:

The standalone version of Spark (without Hadoop-HDFS and YARN) works perfectly. The master
starts and also the worker starts and get registered at the master.

Logfile of the master:

Spark Command: /usr/lib/jvm/jdk//bin/java -cp :/home/thorsten/spark/conf:/home/thorsten/spark/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop1.0.4.jar
-Djava.library.path= -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip base --port
7077 --webui-port 8080
========================================

13/11/01 18:51:14 INFO Slf4jEventHandler: Slf4jEventHandler started
13/11/01 18:51:14 INFO Master: Starting Spark master at spark://base:7077
13/11/01 18:51:14 INFO MasterWebUI: Started Master web UI at http://base:8080
13/11/01 18:51:17 INFO Master: Registering worker base:32942 with 1 cores, 512.0 MB RAM


Because I want to try out Spark with HDFS too, I tried to compile it also that way:


SPARK_HADOOP_VERSION=2.2.0 sbt/sbt assemblyBut with this compiled version, the registration
of workers at the master is not working anymore.

Here is the logfile of the slave:

Spark Command: java -cp :/home/thorsten/spark-hdfs/conf:/home/thorsten/spark-hdfs/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.2.0.jar
-Djava.library.path= -Xms512m -Xmx512m org.apache.spark.deploy.worker.Worker spark://base:7077
--webui-port 8081
========================================

13/11/01 18:49:57 INFO Slf4jEventHandler: Slf4jEventHandler started
13/11/01 18:49:57 INFO Worker: Starting Spark worker base:37950 with 1 cores, 512.0 MB RAM
13/11/01 18:49:57 INFO Worker: Spark home: /home/thorsten/spark-hdfs
13/11/01 18:49:57 INFO WorkerWebUI: Started Worker web UI at http://base:8081
13/11/01 18:49:57 INFO Worker: Connecting to master spark://base:7077
13/11/01 18:49:58 ERROR Worker: Connection to master failed! Shutting down.

The logfile of the master:

Spark Command: /usr/lib/jvm/jdk//bin/java -cp :/home/thorsten/spark-hdfs/conf:/home/thorsten/spark-hdfs/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.2.0.jar
-Djava.library.path= -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip base --port
7077 --webui-port 8080
========================================

13/11/01 18:49:55 INFO Slf4jEventHandler: Slf4jEventHandler started
13/11/01 18:49:55 INFO Master: Starting Spark master at spark://base:7077
13/11/01 18:49:55 INFO MasterWebUI: Started Master web UI at http://base:8080

Hope anybody could help me, solving that problem.

Thanks
Thorsten
Mime
View raw message