spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: run spark0.9.1 on yarn with hadoop CDH4
Date Sat, 17 May 2014 01:43:52 GMT
Hi Sophia,

Unfortunately, Spark doesn't work against YARN in CDH4.  The YARN APIs
changed quite a bit before finally being stabilized in Hadoop 2.2 and CDH5.
 Spark on YARN supports Hadoop 0.23.* and Hadoop 2.2+ / CDH5.0+, but does
not support CDH4, which is somewhere in between.

-Sandy


On Fri, May 9, 2014 at 12:13 AM, Arpit Tak <arpit.t@sigmoidanalytics.com>wrote:

> Also try this out , we have already done this ..
> It will help you..
>
> http://docs.sigmoidanalytics.com/index.php/Setup_hadoop_2.0.0-cdh4.2.0_and_spark_0.9.0_on_ubuntu_12.04
>
>
>
>
> On Tue, May 6, 2014 at 10:17 PM, Andrew Lee <alee526@hotmail.com> wrote:
>
>> Please check JAVA_HOME. Usually it should point to /usr/java/default on
>> CentOS/Linux.
>>
>> or FYI: http://stackoverflow.com/questions/1117398/java-home-directory
>>
>>
>> > Date: Tue, 6 May 2014 00:23:02 -0700
>> > From: sln-1026@163.com
>> > To: user@spark.incubator.apache.org
>> > Subject: run spark0.9.1 on yarn with hadoop CDH4
>>
>> >
>> > Hi all,
>> > I have make HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory
>> which
>> > contains the (client side) configuration files for the hadoop cluster.
>> > The command to launch the YARN Client which I run is like this:
>> >
>> > #
>> >
>> SPARK_JAR=./~/spark-0.9.1/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
>> > ./bin/spark-class org.apache.spark.deploy.yarn.Client\--jar
>> >
>> examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar\--class
>> > org.apache.spark.examples.SparkPi\--args yarn-standalone \--num-workers
>> 3
>> > \--master-memory 2g \--worker-memory 2g \--worker-cores 1
>> > ./bin/spark-class: line 152: /usr/lib/jvm/java-7-sun/bin/java: No such
>> file
>> > or directory
>> > ./bin/spark-class: line 152: exec: /usr/lib/jvm/java-7-sun/bin/java:
>> cannot
>> > execute: No such file or directory
>> > How to make it runs well?
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/run-spark0-9-1-on-yarn-with-hadoop-CDH4-tp5426.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>

Mime
View raw message