spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Bender <jonathan.ben...@gmail.com>
Subject Spark / YARN classpath issues
Date Thu, 22 May 2014 18:38:10 GMT
Hey all,

I'm working through the basic SparkPi example on a YARN cluster, and i'm
wondering why my containers don't pick up the spark assembly classes.

I built the latest spark code against CDH5.0.0

Then ran the following:
SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar
\
      ./bin/spark-class org.apache.spark.deploy.yarn.Client \
      --jar
examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar
\
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1

The job dies, and in the stderr from the containers I see
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/deploy/yarn/ApplicationMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.ApplicationMaster
 at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)

my yarn-site.xml contains the following classpath:
  <property>
    <name>yarn.application.classpath</name>
    <value>
    /etc/hadoop/conf/,
    /usr/lib/hadoop/*,/usr/lib/hadoop//lib/*,
    /usr/lib/hadoop-hdfs/*,/user/lib/hadoop-hdfs/lib/*,
    /usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*,
    /usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,
    /usr/lib/avro/*
    </value>
  </property>

I've confirmed that the spark-assembly JAR has this class.  Does it
actually need to be defined in yarn.application.classpath or should the
spark client take care of ensuring the necessary JARs are added during job
submission?

Any tips would be greatly appreciated!
Cheers,
Jon

Mime
View raw message