spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dong Lei <dong...@microsoft.com>
Subject ClassNotDefException when using spark-submit with multiple jars and files located on HDFS
Date Tue, 09 Jun 2015 03:35:31 GMT
Hi, spark-users:

I'm using spark-submit to submit multiple jars and files(all in HDFS) to run a job, with the
following command:

Spark-submit
  --class myClass
 --master spark://localhost:7077/
  --deploy-mode cluster
  --jars hdfs://localhost/1.jar, hdfs://localhost/2.jar
  --files hdfs://localhost/1.txt, hdfs://localhost/2.txt
 hdfs://localhost/main.jar

the stderr in the driver showed java.lang.ClassNotDefException for a class in 1.jar.

I checked the log that spark has added these jars:
     INFO SparkContext: Added JAR hdfs:// ...1.jar
     INFO SparkContext: Added JAR hdfs:// ...2.jar

In the folder of the driver, I only saw the main.jar is copied to that place, but  the other
jars and files were not there

Could someone explain how should I pass the jars and files needed by the main jar to spark?

If my class in main.jar refer to these files with a relative path, will spark copy these files
into one folder?

BTW, my class works in a client mode with all jars and files in local.

Thanks
Dong Lei

Mime
View raw message