spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhihang Fan <canoe...@gmail.com>
Subject Re: Can Spark 1.0.2 run on CDH-4.3.0 with yarn? And Will Spark 1.2.0 support CDH5.1.2 with yarn?
Date Thu, 18 Dec 2014 09:44:58 GMT
Hi, Sean
   Thank you for your reply. I will try to use Spark 1.1 and 1.2 on CHD5.X.
 :)


2014-12-18 17:38 GMT+08:00 Sean Owen <sowen@cloudera.com>:
>
> The question is really: will Spark 1.1 work with a particular version
> of YARN? many, but not all versions of YARN are supported. The
> "stable" versions are (2.2.x+). Before that, support is patchier, and
> in fact has been removed in Spark 1.3.
>
> The "yarn" profile supports "YARN stable" which is about 2.2.x and
> onwards. The "yarn-alpha" profile should work for YARN about 0.23.x.
> 2.0.x and 2.1.x were a sort of "beta" period and I recall that
> yarn-alpha works with some of it, but not all, and there is no
> yarn-beta profile.
>
> I believe early CDH 4.x has basically "YARN beta". Later 4.x has
> stable. I think I'd try the yarn-alpha profile and see if it compiles.
> But the version of YARN in that release may well be among those that
> fall in the gap between "alpha" and "stable" support.
>
> Thankfully things got a lot more stable past Hadoop / YARN 2.2 or so,
> so it far more just works without version issues. And CDH 5 is based
> on Hadoop 2.3 and then 2.5, so you should be much more able to build
> whatever versions together that you want.
>
> CDH 5.1.x ships Spark 1.0.x. There should be no problem using 1.1.x,
> 1.2.x, etc. with it; you just need to make and support your own
> binaries. 5.2.x has 1.1.x; 5.3.x will have 1.2.x.
>
> On Thu, Dec 18, 2014 at 9:18 AM, Canoe <canoefzh@gmail.com> wrote:
> > I did not compile spark 1.1.0 source code on CDH4.3.0 with yarn
> successfully.
> > Does it support CDH4.3.0 with yarn ?
> > And will spark 1.2.0 support CDH5.1.2?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Can-Spark-1-0-2-run-on-CDH-4-3-0-with-yarn-And-Will-Spark-1-2-0-support-CDH5-1-2-with-yarn-tp20760.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>


-- 
谁谓河广,一苇航之

Mime
View raw message