spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: spark-0.8.0 and hadoop-2.1.0-beta
Date Tue, 29 Oct 2013 18:34:35 GMT
I’m curious, Viren, do you have a patch you could post to build this against YARN 2.1 / 2.2?
It would be nice to see how big the changes are.

Matei

On Sep 30, 2013, at 10:14 AM, viren kumar <virensk@gmail.com> wrote:

> I was able to get Spark 0.8.0 to compile with Hadoop/Yarn 2.1.0-beta, by following some
of the changes described here: http://hortonworks.com/blog/stabilizing-yarn-apis-for-apache-hadoop-2-beta-and-beyond/
> 
> That should help you build most of it. One change not covered there is the change from
ProtoUtils.convertFromProtoFormat(containerToken, cmAddress) to ConverterUtils.convertFromYarn(containerToken,
cmAddress).
> 
> Not 100% sure that my changes are correct. 
> 
> Hope that helps,
> Viren
> 
> 
> On Sun, Sep 29, 2013 at 8:59 AM, Matei Zaharia <matei.zaharia@gmail.com> wrote:
> Hi Terence,
> 
> YARN's API changed in an incompatible way in Hadoop 2.1.0, so I'd suggest sticking with
2.0.x for now. We may create a different branch for this version. Unfortunately due to the
API change it may not be possible to support this version while also supporting other widely-used
versions like 0.23.x.
> 
> Matei
> 
> On Sep 29, 2013, at 11:00 AM, Terance Dias <terance.dias@gmail.com> wrote:
> 
> >
> > Hi, I'm trying to build spark-0.8.0 with hadoop-2.1.0-beta.
> > I have changed the following properties in SparkBuild.scala file.
> >
> > val DEFAULT_HADOOP_VERSION = "2.1.0-beta"
> > val DEFAULT_YARN = true
> >
> > when i do sbt clean compile, I get an error saying
> >
> > [error] /usr/local/spark-0.8.0-incubating/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:42:
not found: type AMRMProtocol
> > [error]   private var resourceManager: AMRMProtocol = null
> >
> > Thanks,
> > Terance.
> >
> 
> 


Mime
View raw message