spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jenny Zhao <linlin200...@gmail.com>
Subject configure spark history server for running on Yarn
Date Fri, 02 May 2014 00:09:20 GMT
Hi,

I have installed spark 1.0 from the branch-1.0, build went fine, and I have
tried running the example on Yarn client mode, here is my command:

/home/hadoop/spark-branch-1.0/bin/spark-submit
/home/hadoop/spark-branch-1.0/examples/target/scala-2.10/spark-examples-1.0.0-hadoop2.2.0.jar
--master yarn --deploy-mode client --executor-memory 6g --executor-cores 3
--driver-memory 3g --name SparkPi --num-executors 2 --class
org.apache.spark.examples.SparkPi yarn-client 5

after the run, I was not being able to retrieve the log from Yarn's web UI,
while I have tried to specify the history server in spark-env.sh

export SPARK_DAEMON_JAVA_OPTS="-Dspark.yarn.historyServer.address=
master:18080 <http://hdtest022.svl.ibm.com:18080>"


I also tried to specify it in spark-defaults.conf, doesn't work as well, I
would appreciate if someone can tell me what is the way of specifying it
either in spark-env.sh or spark-defaults.conf, so that this option can be
applied to any spark application.

another thing I found is the usage output for spark-submit is not
complete/not in sync with the online documentation, hope it is addressed
with the formal release.

and is this the latest documentation for spark 1.0?
http://people.csail.mit.edu/matei/spark-unified-docs/running-on-yarn.html

Thank you!

Mime
View raw message