spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hiroyuki Yamada <>
Subject spark-submit with cluster deploy mode fails with ClassNotFoundException (jars are not passed around properley?)
Date Sat, 12 Mar 2016 06:23:23 GMT

I am trying to work with spark-submit with cluster deploy mode in single
but I keep getting ClassNotFoundException as shown below.
(in this case, snakeyaml.jar is not found from the spark cluster)


16/03/12 14:19:12 INFO Remoting: Starting remoting
16/03/12 14:19:12 INFO Remoting: Remoting started; listening on
addresses :[akka.tcp://Driver@]
16/03/12 14:19:12 INFO util.Utils: Successfully started service
'Driver' on port 52993.
16/03/12 14:19:12 INFO worker.WorkerWatcher: Connecting to worker
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(
	at java.lang.reflect.Method.invoke(
	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NoClassDefFoundError: org/yaml/snakeyaml/Yaml
	... 6 more
Caused by: java.lang.ClassNotFoundException: org.yaml.snakeyaml.Yaml
	at java.lang.ClassLoader.findClass(
	at org.apache.spark.util.ParentClassLoader.findClass(ParentClassLoader.scala:26)
	at java.lang.ClassLoader.loadClass(
	at org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:34)
	at org.apache.spark.util.ChildFirstURLClassLoader.liftedTree1$1(MutableURLClassLoader.scala:75)
	at org.apache.spark.util.ChildFirstURLClassLoader.loadClass(MutableURLClassLoader.scala:71)
	at java.lang.ClassLoader.loadClass(
	... 9 more
16/03/12 14:19:12 INFO util.Utils: Shutdown hook called


I can submit a job successfully with client mode, but I can't with cluster
so, it is a matter of not properly passing jars (snakeyaml) to the cluster.

The actual command I tried is:

$ spark-submit --master spark:// --deploy-mode cluster
--jars all-the-jars(with comma separated) --class analytics.jar
(of course, snakeyaml.jar is specified after --jars)

I tried spark.executor.extraClassPath and spark.driver.extraClassPath in
spark-defaults.conf to specifiy snakeyaml.jar,
but none of those worked.

I also found couple of similar issues posted in the mailing list or other
but, it is not responded back properly or it didn't work to me.


Could anyone give me a help ?

Best regards,

View raw message