spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yash Sharma <yash...@gmail.com>
Subject Re: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher
Date Wed, 22 Jun 2016 06:46:12 GMT
Are you able to run anything else on the cluster, I suspect its yarn that
not able to run the class. If you could just share the logs in pastebin we
could confirm that.

On Wed, Jun 22, 2016 at 4:43 PM, 另一片天 <958943172@qq.com> wrote:

> i  want to avoid Uploading resource file (especially jar package),because
> them very big,the application will wait for too long,there are good method??
> so i config that para, but not get the my want to effect。
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Yash Sharma";<yash360@gmail.com>;
> *发送时间:* 2016年6月22日(星期三) 下午2:34
> *收件人:* "另一片天"<958943172@qq.com>;
> *抄送:* "Saisai Shao"<sai.sai.shao@gmail.com>; "user"<user@spark.apache.org>;
>
> *主题:* Re: Could not find or load main class
> org.apache.spark.deploy.yarn.ExecutorLauncher
>
> Try with : --master yarn-cluster
>
> On Wed, Jun 22, 2016 at 4:30 PM, 另一片天 <958943172@qq.com> wrote:
>
>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>> yarn-client --driver-memory 512m --num-executors 2 --executor-memory 512m
>> --executor-cores 2
>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
>> 10
>> Warning: Skip remote jar
>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar.
>> java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:348)
>> at org.apache.spark.util.Utils$.classForName(Utils.scala:174)
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "Yash Sharma";<yash360@gmail.com>;
>> *发送时间:* 2016年6月22日(星期三) 下午2:28
>> *收件人:* "另一片天"<958943172@qq.com>;
>> *抄送:* "Saisai Shao"<sai.sai.shao@gmail.com>; "user"<user@spark.apache.org>;
>>
>> *主题:* Re: Could not find or load main class
>> org.apache.spark.deploy.yarn.ExecutorLauncher
>>
>> Or better , try the master as yarn-cluster,
>>
>> ./bin/spark-submit \
>> --class org.apache.spark.examples.SparkPi \
>> --master yarn-cluster \
>> --driver-memory 512m \
>> --num-executors 2 \
>> --executor-memory 512m \
>> --executor-cores 2 \
>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.
>> 1-hadoop2.6.0.jar
>>
>> On Wed, Jun 22, 2016 at 4:27 PM, 另一片天 <958943172@qq.com> wrote:
>>
>>> Is it able to run on local mode ?
>>>
>>> what mean?? standalone mode ?
>>>
>>>
>>> ------------------ 原始邮件 ------------------
>>> *发件人:* "Yash Sharma";<yash360@gmail.com>;
>>> *发送时间:* 2016年6月22日(星期三) 下午2:18
>>> *收件人:* "Saisai Shao"<sai.sai.shao@gmail.com>;
>>> *抄送:* "另一片天"<958943172@qq.com>; "user"<user@spark.apache.org>;
>>> *主题:* Re: Could not find or load main class
>>> org.apache.spark.deploy.yarn.ExecutorLauncher
>>>
>>> Try providing the jar with the hdfs prefix. Its probably just because
>>> its not able to find the jar on all nodes.
>>>
>>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.
>>> 1-hadoop2.6.0.jar
>>>
>>> Is it able to run on local mode ?
>>>
>>> On Wed, Jun 22, 2016 at 4:14 PM, Saisai Shao <sai.sai.shao@gmail.com>
>>> wrote:
>>>
>>>> spark.yarn.jar (none) The location of the Spark jar file, in case
>>>> overriding the default location is desired. By default, Spark on YARN will
>>>> use a Spark jar installed locally, but the Spark jar can also be in a
>>>> world-readable location on HDFS. This allows YARN to cache it on nodes so
>>>> that it doesn't need to be distributed each time an application runs. To
>>>> point to a jar on HDFS, for example, set this configuration to
>>>> hdfs:///some/path.
>>>>
>>>> spark.yarn.jar is used for spark run-time system jar, which is spark
>>>> assembly jar, not the application jar (example-assembly jar). So in your
>>>> case you upload the example-assembly jar into hdfs, in which spark system
>>>> jars are not packed, so ExecutorLaucher cannot be found.
>>>>
>>>> Thanks
>>>> Saisai
>>>>
>>>> On Wed, Jun 22, 2016 at 2:10 PM, 另一片天 <958943172@qq.com> wrote:
>>>>
>>>>> shihj@master:/usr/local/spark/spark-1.6.1-bin-hadoop2.6$
>>>>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>>>>> yarn-client --driver-memory 512m --num-executors 2 --executor-memory
512m
>>>>> --executor-cores 2
>>>>> /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar 10
>>>>> Warning: Local jar
>>>>> /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar does not exist,
>>>>> skipping.
>>>>> java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>> at java.lang.Class.forName0(Native Method)
>>>>> at java.lang.Class.forName(Class.java:348)
>>>>> at org.apache.spark.util.Utils$.classForName(Utils.scala:174)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>> get error at once
>>>>> ------------------ 原始邮件 ------------------
>>>>> *发件人:* "Yash Sharma";<yash360@gmail.com>;
>>>>> *发送时间:* 2016年6月22日(星期三) 下午2:04
>>>>> *收件人:* "另一片天"<958943172@qq.com>;
>>>>> *抄送:* "user"<user@spark.apache.org>;
>>>>> *主题:* Re: Could not find or load main class
>>>>> org.apache.spark.deploy.yarn.ExecutorLauncher
>>>>>
>>>>> How about supplying the jar directly in spark submit -
>>>>>
>>>>> ./bin/spark-submit \
>>>>>> --class org.apache.spark.examples.SparkPi \
>>>>>> --master yarn-client \
>>>>>> --driver-memory 512m \
>>>>>> --num-executors 2 \
>>>>>> --executor-memory 512m \
>>>>>> --executor-cores 2 \
>>>>>> /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
>>>>>
>>>>>
>>>>> On Wed, Jun 22, 2016 at 3:59 PM, 另一片天 <958943172@qq.com>
wrote:
>>>>>
>>>>>> i  config this  para  at spark-defaults.conf
>>>>>> spark.yarn.jar
>>>>>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
>>>>>>
>>>>>> then ./bin/spark-submit --class org.apache.spark.examples.SparkPi
>>>>>> --master yarn-client --driver-memory 512m --num-executors 2
>>>>>> --executor-memory 512m --executor-cores 2    10:
>>>>>>
>>>>>>
>>>>>>
>>>>>>    - Error: Could not find or load main class
>>>>>>    org.apache.spark.deploy.yarn.ExecutorLauncher
>>>>>>
>>>>>> but  i don't config that para ,there no error  why???that para is
>>>>>> only avoid Uploading resource file(jar package)??
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message