spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jenny Zhao <>
Subject configure spark history server for running on Yarn
Date Fri, 02 May 2014 00:09:20 GMT

I have installed spark 1.0 from the branch-1.0, build went fine, and I have
tried running the example on Yarn client mode, here is my command:

--master yarn --deploy-mode client --executor-memory 6g --executor-cores 3
--driver-memory 3g --name SparkPi --num-executors 2 --class
org.apache.spark.examples.SparkPi yarn-client 5

after the run, I was not being able to retrieve the log from Yarn's web UI,
while I have tried to specify the history server in

export SPARK_DAEMON_JAVA_OPTS="-Dspark.yarn.historyServer.address=
master:18080 <>"

I also tried to specify it in spark-defaults.conf, doesn't work as well, I
would appreciate if someone can tell me what is the way of specifying it
either in or spark-defaults.conf, so that this option can be
applied to any spark application.

another thing I found is the usage output for spark-submit is not
complete/not in sync with the online documentation, hope it is addressed
with the formal release.

and is this the latest documentation for spark 1.0?

Thank you!

View raw message