spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Haviv <danielru...@gmail.com>
Subject Re: Could not find the main class: org.apache.spark.deploy.SparkSubmit
Date Tue, 16 Dec 2014 17:22:24 GMT
Completely diffrent than the one I set:
Classpath is
 ::/tmp/spark/spark-branch-1.1/conf:/tmp/spark/spark-branch-1.1/assembly/target/scala-2.10/spark-assembly-1.1.2-SNAPSHOT-hadoop2.3.0.jar:/tmp/spark/spark-branch-1.1/lib_managed/jars/datanucleus-core-3.2.2.jar:/tmp/spark/spark-branch-1.1/lib_managed/jars/datanucleus-rdbms-3.2.1.jar:/tmp/spark/spark-branch-1.1/lib_managed/jars/datanucleus-api-jdo-3.2.1.jar



On Tue, Dec 16, 2014 at 7:18 PM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:
>
> Can you open the file bin/spark-class and then put an echo $CLASSPATH
> below the place where they exports it and see what are the contents?
> On 16 Dec 2014 22:46, "Daniel Haviv" <danielrulez@gmail.com> wrote:
>
>> I've added every jar in the lib dir to my classpath and still no luck:
>>
>>
>> CLASSPATH=/tmp/spark/spark-branch-1.1/lib/datanucleus-api-jdo-3.2.1.jar:/tmp/spark/spark-branch-1.1/lib/datanucleus-core-3.2.2.jar:/tmp/spark/spark-branch-1.1/lib/datanucleus-rdbms-3.2.1.jar:/tmp/spark/spark-branch-1.1/lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.3.0.jar:/tmp/spark/spark-branch-1.1/lib/spark-bagel_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-bagel_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-catalyst_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-catalyst_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-catalyst_2.10-1.1.2-SNAPSHOT-tests.jar:/tmp/spark/spark-branch-1.1/lib/spark-core_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-core_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-examples-1.1.2-SNAPSHOT-hadoop2.3.0.jar:/tmp/spark/spark-branch-1.1/lib/spark-examples_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-examples_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-graphx_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-graphx_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-hive_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-hive_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-hive-thriftserver_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-hive-thriftserver_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-mllib_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-mllib_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-repl_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-repl_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-sql_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-sql_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming_2.10-1.1.2-SNAPSHOT-tests.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-flume_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-flume_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-flume-sink_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-flume-sink_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-kafka_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-kafka_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-mqtt_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-mqtt_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-twitter_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-twitter_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-zeromq_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-zeromq_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-tools_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-tools_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-yarn_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-yarn_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/TestSerDe.jar
>>
>>
>>
>>
>> On Tue, Dec 16, 2014 at 7:05 PM, Akhil Das <akhil@sigmoidanalytics.com>
>> wrote:
>>>
>>> And this is how my classpath looks like
>>>
>>> Spark assembly has been built with Hive, including Datanucleus jars on
>>> classpath
>>>
>>> ::/home/akhld/mobi/localcluster/spark-1/conf:/home/akhld/mobi/localcluster/spark-1/lib/spark-assembly-1.1.0-hadoop1.0.4.jar:/home/akhld/mobi/localcluster/spark-1/lib/datanucleus-core-3.2.2.jar:/home/akhld/mobi/localcluster/spark-1/lib/datanucleus-rdbms-3.2.1.jar:/home/akhld/mobi/localcluster/spark-1/lib/datanucleus-api-jdo-3.2.1.jar
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Tue, Dec 16, 2014 at 10:33 PM, Daniel Haviv <danielrulez@gmail.com>
>>> wrote:
>>>>
>>>> Same here...
>>>> # jar tf lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.3.0.jar  | grep
>>>> SparkSubmit.class
>>>> *org/apache/spark/deploy/SparkSubmit.class*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Dec 16, 2014 at 6:50 PM, Akhil Das <akhil@sigmoidanalytics.com>
>>>> wrote:
>>>>>
>>>>> This is how it looks on my machine.
>>>>>
>>>>> [image: Inline image 1]
>>>>>
>>>>> Thanks
>>>>> Best Regards
>>>>>
>>>>> On Tue, Dec 16, 2014 at 9:33 PM, Daniel Haviv <danielrulez@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> That's the first thing I tried... still the same error:
>>>>>> hdfs@ams-rsrv01:~$ export CLASSPATH=/tmp/spark/spark-branch-1.1/lib
>>>>>> hdfs@ams-rsrv01:~$ cd /tmp/spark/spark-branch-1.1
>>>>>> hdfs@ams-rsrv01:/tmp/spark/spark-branch-1.1$ ./bin/spark-shell
>>>>>>
>>>>>> Spark assembly has been built with Hive, including Datanucleus jars
>>>>>> on classpath
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/spark/deploy/SparkSubmit
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.spark.deploy.SparkSubmit
>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
>>>>>>         at
>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
>>>>>> Could not find the main class: org.apache.spark.deploy.SparkSubmit.
>>>>>> Program will exit.
>>>>>>
>>>>>> Thanks,
>>>>>> Daniel
>>>>>>
>>>>>> On Tue, Dec 16, 2014 at 1:53 PM, Akhil Das <
>>>>>> akhil@sigmoidanalytics.com> wrote:
>>>>>>>
>>>>>>> print the CLASSPATH and make sure the spark assembly jar is there
in
>>>>>>> the classpath
>>>>>>>
>>>>>>> Thanks
>>>>>>> Best Regards
>>>>>>>
>>>>>>> On Tue, Dec 16, 2014 at 5:04 PM, Daniel Haviv <danielrulez@gmail.com
>>>>>>> > wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> I've built spark successfully with maven but when I try to
run
>>>>>>>> spark-shell I get the following errors:
>>>>>>>>
>>>>>>>> Spark assembly has been built with Hive, including Datanucleus
jars
>>>>>>>> on classpath
>>>>>>>>
>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>>> org/apache/spark/deploy/SparkSubmit
>>>>>>>>
>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>> org.apache.spark.deploy.SparkSubmit
>>>>>>>>
>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>>>>>>
>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>
>>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>
>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
>>>>>>>>
>>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)at
>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:268)Could
not
>>>>>>>> find the main class: org.apache.spark.deploy.SparkSubmit.
Program will exit.
>>>>>>>> What can be wrong?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>

Mime
View raw message