spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yash Sharma <yash...@gmail.com>
Subject Re: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher
Date Wed, 22 Jun 2016 07:06:04 GMT
I cannot get a lot of info from these logs but it surely seems like yarn
setup issue. Did you try the local mode to check if it works -

./bin/spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master local[4] \
> spark-examples-1.6.1-hadoop2.6.0.jar 10


Note - the jar is a local one

On Wed, Jun 22, 2016 at 4:50 PM, 另一片天 <958943172@qq.com> wrote:

>  Application application_1466568126079_0006 failed 2 times due to AM
> Container for appattempt_1466568126079_0006_000002 exited with exitCode: 1
> For more detailed output, check application tracking page:
> http://master:8088/proxy/application_1466568126079_0006/Then, click on
> links to logs of each attempt.
> Diagnostics: Exception from container-launch.
> Container id: container_1466568126079_0006_02_000001
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> Failing this attempt. Failing the application.
>
> but command get error
>
> shihj@master:~/workspace/hadoop-2.6.4$ yarn logs -applicationId
> application_1466568126079_0006
> Usage: yarn [options]
>
> yarn: error: no such option: -a
>
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Yash Sharma";<yash360@gmail.com>;
> *发送时间:* 2016年6月22日(星期三) 下午2:46
> *收件人:* "另一片天"<958943172@qq.com>;
> *抄送:* "Saisai Shao"<sai.sai.shao@gmail.com>; "user"<user@spark.apache.org>;
>
> *主题:* Re: Could not find or load main class
> org.apache.spark.deploy.yarn.ExecutorLauncher
>
> Are you able to run anything else on the cluster, I suspect its yarn that
> not able to run the class. If you could just share the logs in pastebin we
> could confirm that.
>
> On Wed, Jun 22, 2016 at 4:43 PM, 另一片天 <958943172@qq.com> wrote:
>
>> i  want to avoid Uploading resource file (especially jar package),because
>> them very big,the application will wait for too long,there are good method??
>> so i config that para, but not get the my want to effect。
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "Yash Sharma";<yash360@gmail.com>;
>> *发送时间:* 2016年6月22日(星期三) 下午2:34
>> *收件人:* "另一片天"<958943172@qq.com>;
>> *抄送:* "Saisai Shao"<sai.sai.shao@gmail.com>; "user"<user@spark.apache.org>;
>>
>> *主题:* Re: Could not find or load main class
>> org.apache.spark.deploy.yarn.ExecutorLauncher
>>
>> Try with : --master yarn-cluster
>>
>> On Wed, Jun 22, 2016 at 4:30 PM, 另一片天 <958943172@qq.com> wrote:
>>
>>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>>> yarn-client --driver-memory 512m --num-executors 2 --executor-memory 512m
>>> --executor-cores 2
>>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
>>> 10
>>> Warning: Skip remote jar
>>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar.
>>> java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:348)
>>> at org.apache.spark.util.Utils$.classForName(Utils.scala:174)
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>>
>>>
>>> ------------------ 原始邮件 ------------------
>>> *发件人:* "Yash Sharma";<yash360@gmail.com>;
>>> *发送时间:* 2016年6月22日(星期三) 下午2:28
>>> *收件人:* "另一片天"<958943172@qq.com>;
>>> *抄送:* "Saisai Shao"<sai.sai.shao@gmail.com>; "user"<
>>> user@spark.apache.org>;
>>> *主题:* Re: Could not find or load main class
>>> org.apache.spark.deploy.yarn.ExecutorLauncher
>>>
>>> Or better , try the master as yarn-cluster,
>>>
>>> ./bin/spark-submit \
>>> --class org.apache.spark.examples.SparkPi \
>>> --master yarn-cluster \
>>> --driver-memory 512m \
>>> --num-executors 2 \
>>> --executor-memory 512m \
>>> --executor-cores 2 \
>>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.
>>> 1-hadoop2.6.0.jar
>>>
>>> On Wed, Jun 22, 2016 at 4:27 PM, 另一片天 <958943172@qq.com> wrote:
>>>
>>>> Is it able to run on local mode ?
>>>>
>>>> what mean?? standalone mode ?
>>>>
>>>>
>>>> ------------------ 原始邮件 ------------------
>>>> *发件人:* "Yash Sharma";<yash360@gmail.com>;
>>>> *发送时间:* 2016年6月22日(星期三) 下午2:18
>>>> *收件人:* "Saisai Shao"<sai.sai.shao@gmail.com>;
>>>> *抄送:* "另一片天"<958943172@qq.com>; "user"<user@spark.apache.org>;
>>>> *主题:* Re: Could not find or load main class
>>>> org.apache.spark.deploy.yarn.ExecutorLauncher
>>>>
>>>> Try providing the jar with the hdfs prefix. Its probably just because
>>>> its not able to find the jar on all nodes.
>>>>
>>>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.
>>>> 1-hadoop2.6.0.jar
>>>>
>>>> Is it able to run on local mode ?
>>>>
>>>> On Wed, Jun 22, 2016 at 4:14 PM, Saisai Shao <sai.sai.shao@gmail.com>
>>>> wrote:
>>>>
>>>>> spark.yarn.jar (none) The location of the Spark jar file, in case
>>>>> overriding the default location is desired. By default, Spark on YARN
will
>>>>> use a Spark jar installed locally, but the Spark jar can also be in a
>>>>> world-readable location on HDFS. This allows YARN to cache it on nodes
so
>>>>> that it doesn't need to be distributed each time an application runs.
To
>>>>> point to a jar on HDFS, for example, set this configuration to
>>>>> hdfs:///some/path.
>>>>>
>>>>> spark.yarn.jar is used for spark run-time system jar, which is spark
>>>>> assembly jar, not the application jar (example-assembly jar). So in your
>>>>> case you upload the example-assembly jar into hdfs, in which spark system
>>>>> jars are not packed, so ExecutorLaucher cannot be found.
>>>>>
>>>>> Thanks
>>>>> Saisai
>>>>>
>>>>> On Wed, Jun 22, 2016 at 2:10 PM, 另一片天 <958943172@qq.com>
wrote:
>>>>>
>>>>>> shihj@master:/usr/local/spark/spark-1.6.1-bin-hadoop2.6$
>>>>>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>>>>>> yarn-client --driver-memory 512m --num-executors 2 --executor-memory
512m
>>>>>> --executor-cores 2
>>>>>> /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar 10
>>>>>> Warning: Local jar
>>>>>> /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar does not
exist,
>>>>>> skipping.
>>>>>> java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>> at org.apache.spark.util.Utils$.classForName(Utils.scala:174)
>>>>>> at
>>>>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
>>>>>> at
>>>>>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>>>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>>>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>> get error at once
>>>>>> ------------------ 原始邮件 ------------------
>>>>>> *发件人:* "Yash Sharma";<yash360@gmail.com>;
>>>>>> *发送时间:* 2016年6月22日(星期三) 下午2:04
>>>>>> *收件人:* "另一片天"<958943172@qq.com>;
>>>>>> *抄送:* "user"<user@spark.apache.org>;
>>>>>> *主题:* Re: Could not find or load main class
>>>>>> org.apache.spark.deploy.yarn.ExecutorLauncher
>>>>>>
>>>>>> How about supplying the jar directly in spark submit -
>>>>>>
>>>>>> ./bin/spark-submit \
>>>>>>> --class org.apache.spark.examples.SparkPi \
>>>>>>> --master yarn-client \
>>>>>>> --driver-memory 512m \
>>>>>>> --num-executors 2 \
>>>>>>> --executor-memory 512m \
>>>>>>> --executor-cores 2 \
>>>>>>> /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 22, 2016 at 3:59 PM, 另一片天 <958943172@qq.com>
wrote:
>>>>>>
>>>>>>> i  config this  para  at spark-defaults.conf
>>>>>>> spark.yarn.jar
>>>>>>> hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar
>>>>>>>
>>>>>>> then ./bin/spark-submit --class org.apache.spark.examples.SparkPi
>>>>>>> --master yarn-client --driver-memory 512m --num-executors 2
>>>>>>> --executor-memory 512m --executor-cores 2    10:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    - Error: Could not find or load main class
>>>>>>>    org.apache.spark.deploy.yarn.ExecutorLauncher
>>>>>>>
>>>>>>> but  i don't config that para ,there no error  why???that para
is
>>>>>>> only avoid Uploading resource file(jar package)??
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message