spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deenar Toraskar <deenar.toras...@gmail.com>
Subject Re: Spark 1.5 on CDH 5.4.0
Date Sun, 01 Nov 2015 14:50:49 GMT
HI guys

I have documented the steps involved in getting Spark 1.5.1 run on CDH
5.4.0 here, let me know if it works for you as well
https://www.linkedin.com/pulse/running-spark-151-cdh-deenar-toraskar-cfa?trk=hp-feed-article-title-publish

Looking forward to CDH 5.5 which supports Spark 1.5.x out of the box.

Regards
Deenar




*Think Reactive Ltd*
deenar.toraskar@thinkreactive.co.uk
07714140812


On 23 October 2015 at 17:31, Deenar Toraskar <deenar.toraskar@gmail.com>
wrote:

> I got this working. For others trying this It turns out in Spark 1.3/CDH5.4
>
> spark.yarn.jar=local:/opt/cloudera/parcels/....
>
> I had changed this to reflect the 1.5.1 version of spark assembly jar
>
> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>
> and this didn't work, I had to drop the "local:" prefix
>
> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>
> Regards
> Deenar
>
> On 23 October 2015 at 17:30, Deenar Toraskar <
> deenar.toraskar@thinkreactive.co.uk> wrote:
>
>> I got this working. For others trying this It turns out in Spark
>> 1.3/CDH5.4
>>
>> spark.yarn.jar=local:/opt/cloudera/parcels/....
>>
>> I had changed this to reflect the 1.5.1 version of spark assembly jar
>>
>> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>>
>> and this didn't work, I had to drop the "local:" prefix
>>
>> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>>
>> Regards
>> Deenar
>>
>>
>>
>>
>> *Think Reactive Ltd*
>> deenar.toraskar@thinkreactive.co.uk
>> 07714140812
>>
>>
>>
>> On 23 October 2015 at 13:34, Deenar Toraskar <deenar.toraskar@gmail.com>
>> wrote:
>>
>>> Sandy
>>>
>>> The assembly jar does contain org.apache.spark.deploy.yarn.ExecutorLauncher.
>>> I am trying to find out how i can increase the logging level, so I know the
>>> exact classpath used by Yarn ContainerLaunch.
>>>
>>> Deenar
>>>
>>> On 23 October 2015 at 03:30, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
>>>
>>>> Hi Deenar,
>>>>
>>>> The version of Spark you have may not be compiled with YARN support.
>>>> If you inspect the contents of the assembly jar, does
>>>> org.apache.spark.deploy.yarn.ExecutorLauncher exist?  If not, you'll
>>>> need to find a version that does have the YARN classes.  You can also build
>>>> your own using the -Pyarn flag.
>>>>
>>>> -Sandy
>>>>
>>>> On Thu, Oct 22, 2015 at 9:04 AM, Deenar Toraskar <
>>>> deenar.toraskar@gmail.com> wrote:
>>>>
>>>>> Hi I have got the prebuilt version of Spark 1.5 for Hadoop 2.6 (
>>>>> http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz)
>>>>> working with CDH 5.4.0 in local mode on a cluster with Kerberos. It works
>>>>> well including connecting to the Hive metastore. I am facing an issue
>>>>> running spark jobs in yarn-client/yarn-cluster mode. The executors fail
to
>>>>> start as java cannot find ExecutorLauncher. Error: Could not find or
>>>>> load main class org.apache.spark.deploy.yarn.ExecutorLauncher client
>>>>> token: N/Adiagnostics: Application application_1443531450011_13437
>>>>> failed 2 times due to AM Container for
>>>>> appattempt_1443531450011_13437_000002 exited with exitCode: 1Stack
>>>>> trace: ExitCodeException exitCode=1:at
>>>>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)at
>>>>> org.apache.hadoop.util.Shell.run(Shell.java:455)at
>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at
>>>>> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)at
>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at
>>>>> java.lang.Thread.run(Thread.java:745) Any ideas as to what might be
>>>>> going wrong. Also how can I turn on more detailed logging to see what
>>>>> command line is being run by Yarn to launch containers? RegardsDeenar
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message