spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hisham Mohamed <Hisham.Moha...@unige.ch>
Subject RE: spark-submit in deployment mode with the "--jars" option
Date Mon, 29 Jun 2015 11:59:52 GMT
Hi Akhil,

Thanks for your reply. 
Here it is

Launch Command: "/usr/lib/jvm/java-7-oracle/jre/bin/java" "-cp" "/etc/spark/:/opt/spark/lib/spark-assembly-1.4.0-hadoop2.3.0.jar:/opt/spark-1.4.0-bin-hadoop2.3/lib/datanucleus-rdbms-3.2.9.jar:/opt/spark-1.4.0-bin
-hadoop2.3/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark-1.4.0-bin-hadoop2.3/lib/datanucleus-core-3.2.10.jar"
"-Xms1024M" "-Xmx1024M" "-Dspark.eventLog.enabled=true" "-Dakka.loglevel=WARNING" "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer"
"-Dspark.executor.memory=1024m" "-Dspark.master=spark://1.2.3.4:7077" "-Dspark.rpc.askTimeout=10"
"-Dspark.app.name=com.scala.algorithm" "-Dspark.driver.supervise=true" "-Dspark.jars=,file:/root/libs/commons-codec-1.2.jar,file:/root/libs/commons-lang3-3.1.jar,file:/root/libs/commons-logging-1.1.1.jar,file:/root/libs/guava-16.0.1.jar,file:/root/libs/httpclient-4.2.5.jar,file:/root/libs/httpcore-4.2.4.jar,file:/root/libs/ivy-2.3.0.jar,file:/root/libs/joda-convert-1.6.jar,file:/root/libs/joda-time-2.3.jar,file:/root/libs/jsr166e-1.1.0.jar,file:/root/libs/kryo-3.0.2.jar,file:/root/libs/lift-json_2.10-2.6.2.jar,file:/root/libs/lz4-1.2.0.jar,file:/root/libs/metrics-core-3.0.2.jar,file:/root/libs/netty-3.9.0.Final.jar,file:/root/libs/reflectasm-1.09.jar,file:/root/libs/slf4j-api-1.7.5.jar,file:/root/libs/snappy-java-1.0.5.jar,file:/root/spark-algorithm-build/0.1.1/algorithm.jar"
"-Dspark.driver.memory=1g" "-XX:MaxPermSize=128m" "org.apache.spark.deploy.worker.DriverWrapper"
"akka.tcp://sparkWorker@1.2.3.4:50911/user/Worker" "/srv/spark/work/driver-20150628164945-0020/analytics-0.1.1.jar"
"com.scala.algorithm"
========================================

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NoClassDefFoundError: net/liftweb/json/DefaultFormats$
	at com.scala.utilities.dal.IOUtil$.parseConfigFile(IOUtil.scala:64)
	at com.scala.algorithmFramework$.main(algorithmfm.scala:20)
	at com.scala.algorithmFramework.main(algorithmfm.scala)
	... 6 more
Caused by: java.lang.ClassNotFoundException: net.liftweb.json.DefaultFormats$
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 9 more



________________________________________
From: Akhil Das [akhil@sigmoidanalytics.com]
Sent: 29 June 2015 09:43
To: Hisham Mohamed
Cc: user@spark.apache.org
Subject: Re: spark-submit in deployment mode with the "--jars" option

Can you paste the stacktrace? Looks like you are missing few jars.

Thanks
Best Regards

On Sun, Jun 28, 2015 at 11:14 PM, hishamm <hisham.mohamed@unige.ch<mailto:hisham.mohamed@unige.ch>>
wrote:
Hi,

I want to deploy my application on a standalone cluster.
Spark submit acts in strange way. When I deploy the application in
*"client"* mode, everything works well and my application can see the
additional jar files.

Here is the command:
>   spark-submit --master spark://1.2.3.4:7077<http://1.2.3.4:7077> --deploy-mode
 client
> --supervise --jars $(echo /myjars/*.jar | tr ' ' ',')  --class
> com.algorithm /my/path/algorithm.jar

However, when I submit the command in *"cluster"* deployment mode. The
driver can not see the additional jars.
I always get *java.lang.ClassNotFoundException*

Here is the command:
>   spark-submit --master spark://1.2.3.4:7077<http://1.2.3.4:7077> --deploy-mode
cluster
> --supervise --jars $(echo /myjars/*.jar | tr ' ' ',')  --class
> com.algorithm /my/path/algorithm.jar


Do I miss something ?

thanks,
Hisham



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-in-deployment-mode-with-the-jars-option-tp23519.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message