spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianshi Huang <jianshi.hu...@gmail.com>
Subject Yarn-client mode and standalone-client mode hang during job start
Date Tue, 17 Jun 2014 09:41:52 GMT
Hi,

I've stuck using either yarn-client or standalone-client mode. Either will
stuck when I submit jobs, the last messages it printed were:

...
14/06/17 02:37:17 INFO spark.SparkContext: Added JAR
file:/x/home/jianshuang/tmp/lib/commons-vfs2.jar at
http://10.196.195.25:56377/jars/commons-vfs2.jar with timestamp
1402997837065
14/06/17 02:37:17 INFO spark.SparkContext: Added JAR
file:/x/home/jianshuang/tmp/rtgraph.jar at
http://10.196.195.25:56377/jars/rtgraph.jar with timestamp 1402997837065
14/06/17 02:37:17 INFO cluster.YarnClusterScheduler: Created
YarnClusterScheduler
14/06/17 02:37:17 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown hook
for context org.apache.spark.SparkContext@6655cf60

I can use yarn-cluster to run my app but it's not very convenient to
monitor the progress.

Standalone-cluster mode doesn't work, it reports file not found error:

Driver successfully submitted as driver-20140617023956-0003
... waiting before polling master for driver state
... polling master for driver state
State of driver-20140617023956-0003 is ERROR
Exception from cluster was: java.io.FileNotFoundException: File
file:/x/home/jianshuang/tmp/rtgraph.jar does not exist


I'm using Spark 1.0.0 and my submit command looks like this:

  ~/spark/spark-1.0.0-hadoop2.4.0/bin/spark-submit --name 'rtgraph' --class
com.paypal.rtgraph.demo.MapReduceWriter --master spark://
lvshdc5en0015.lvs.paypal.com:7077 --jars `find lib -type f | tr '\n' ','`
--executor-memory 20G --total-executor-cores 96 --deploy-mode cluster
rtgraph.jar

List of jars I put in --jars option are:

accumulo-core.jar
accumulo-fate.jar
accumulo-minicluster.jar
accumulo-trace.jar
accumulo-tracer.jar
chill_2.10-0.3.6.jar
commons-math.jar
commons-vfs2.jar
config-1.2.1.jar
gson.jar
guava.jar
joda-convert-1.2.jar
joda-time-2.3.jar
kryo-2.21.jar
libthrift.jar
quasiquotes_2.10-2.0.0-M8.jar
scala-async_2.10-0.9.1.jar
scala-library-2.10.4.jar
scala-reflect-2.10.4.jar


Anyone has hint what went wrong? Really confused.


Cheers,
-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Mime
View raw message