spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "guxiaobo1982" <guxiaobo1...@qq.com>
Subject Re: Can't submit the SparkPi example to local Yarn 2.6.0 installed byambari 1.7.0
Date Mon, 29 Dec 2014 13:41:03 GMT
/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors
3 --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue thequeue lib/spark-examples-1.2.0-hadoop2.6.0.jar
10


Got the same error by the above command, I think I missed the jar containing the Jackson libraries.
 




------------------ Original ------------------
From:  "Sean Owen";<sowen@cloudera.com>;
Send time: Sunday, Dec 28, 2014 3:08 AM
To: ""<guxiaobo1982@qq.com>; 
Cc: "user"<user@spark.apache.org>; 
Subject:  Re: Can't submit the SparkPi example to local Yarn 2.6.0 installed byambari 1.7.0




The problem is a conflicts in the version of Jackson used in your cluster versus what you
run. I would start by taking off things like the assembly jar from your classpath. Try the
userClassPathFirst option as well to avoid using the Jackson in your Hadoop distribution.

 Hi,I build the 1.2.0 version of spark against single node hadoop 2.6.0 installed by ambari
1.7.0, the ./bin/run-example SparkPi 10 command can execute on my local Mac 10.9.5 and the
centos virtual machine, which host hadoop, but I can't run the SparkPi example inside yarn,
it seems there's something wrong with the classpathes:


 
export HADOOP_CONF_DIR=/etc/hadoop/conf
 


 
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors
3 --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue thequeue --jars spark-assembly-1.2.0-hadoop2.6.0.jar,spark-1.2.0-yarn-shuffle.jar,datanucleus-core-3.2.10.jar,datanucleus-rdbms-3.2.9.jar,datanucleus-api-jdo-3.2.6.jar
lib/spark-examples-1.2.0-hadoop2.6.0.jar 10

Spark assembly has been built with Hive, including Datanucleus jars on classpath

14/12/10 15:38:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable

14/12/10 15:39:00 INFO impl.TimelineClientImpl: Timeline service address: http://lix1.bh.com:8188/ws/v1/timeline/

Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jackson/map/deser/std/StdDeserializer

	at java.lang.ClassLoader.defineClass1(Native Method)

	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

	at java.security.AccessController.doPrivileged(Native Method)

	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

	at org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:57)

	at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:47)

	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)

	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:65)

	at org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:501)

	at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)

	at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)

	at org.apache.spark.deploy.yarn.Client.main(Client.scala)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:606)

	at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)

	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)

	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.deser.std.StdDeserializer

	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

	at java.security.AccessController.doPrivileged(Native Method)

	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

	... 28 more

[xiaobogu@lix1 spark-1.2.0-bin-2.6.0]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi
--master yarn-cluster --num-executors 3 --driver-memory 1g --executor-memory 1g --executor-cores
1 --queue thequeue lib/spark-examples-1.2.0-hadoop2.6.0.jar 10

Spark assembly has been built with Hive, including Datanucleus jars on classpath

14/12/10 15:39:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable

14/12/10 15:39:51 INFO impl.TimelineClientImpl: Timeline service address: http://lix1.bh.com:8188/ws/v1/timeline/

Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jackson/map/deser/std/StdDeserializer

	at java.lang.ClassLoader.defineClass1(Native Method)

	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

	at java.security.AccessController.doPrivileged(Native Method)

	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

	at org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:57)

	at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:47)

	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)

	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:65)

	at org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:501)

	at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)

	at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)

	at org.apache.spark.deploy.yarn.Client.main(Client.scala)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:606)

	at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)

	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)

	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.deser.std.StdDeserializer

	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

	at java.security.AccessController.doPrivileged(Native Method)

	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

	... 28 more

 

[xiaobogu@lix1 spark-1.2.0-bin-2.6.0]$
Mime
View raw message