spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sdeb <sangha...@gmail.com>
Subject trouble: Launching spark on hadoop + yarn.
Date Wed, 25 Jun 2014 22:09:22 GMT
i am trying to install spark on Hadoop+Yarn.

I have installed spark using sbt (SPARK_HADOOP_VERSION=2.0.5-alpha
SPARK_YARN=true sbt/sbt assembly ). This has worked fine.

After that I am running :
SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop2.0.5-alpha.jar    
./bin/spark-class org.apache.spark.deploy.yarn.Client       --jar
examples/target/scala-2.10/spark-examples-1.0.0-hadoop2.0.5-alpha.jar      
--class org.apache.spark.examples.SparkPi       --arg yarn-cluster      
--arg 5       --num-executors 3       --driver-memory 4g      
--executor-memory 2g       --executor-cores 1

This process fails with the following error. When I look into the log file,
there is a line containing success,
INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=gpadmin     
IP=10.1.174.109 OPERATION=Start Contai\
ner Request       TARGET=ContainerManageImpl      RESULT=SUCCESS 
APPID=application_1399675492314_0012   
CONTAINERID=container_1399675492314_\
0012_01_000010

 & several lines containing failure,

2014-06-13 18:08:36,661 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=UnknownUser 
IP=10.1.174.105 OPERATION=Stop Contain\
er Request        TARGET=ContainerManagerImpl     RESULT=FAILURE 
DESCRIPTION=Trying to stop unknown container!  
APPID=application_1399675492\
314_0012    CONTAINERID=container_1399675492314_0012_01_000001

2014-06-25 11:57:40,190 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=UnknownUser 
IP=10.1.174.105 OPERATION=Stop Contain\
er Request        TARGET=ContainerManagerImpl     RESULT=FAILURE 
DESCRIPTION=Trying to stop unknown container!  
APPID=application_1399675492\
314_0025    CONTAINERID=container_1399675492314_0025_01_000001

2014-06-25 12:29:52,679 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1399675492314\
_0026_01_000003 transitioned from RUNNING to EXITED_WITH_FAILURE

2014-06-25 12:29:52,698 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=gpadmin     
OPERATION=Container Finished - Failed \
  TARGET=ContainerImpl    RESULT=FAILURE  DESCRIPTION=Container failed with
state: EXITED_WITH_FAILURE    APPID=application_1399675492314_0026\
    CONTAINERID=container_1399675492314_0026_01_000003
2014-06-25 12:29:52,698 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1399675492314\
_0026_01_000003 transitioned from EXITED_WITH_FAILURE to DONE

& the finally I get:
Failed while trying to construct the redirect url to the log server. Log
Server url may not be configured. 

Do I need to change the yarn-site.xml file to configure a log server? If so
how?

--Sanghamitra.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/trouble-Launching-spark-on-hadoop-yarn-tp8297.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message