spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liu, Raymond" <>
Subject RE: spark-0.8.0 and hadoop-2.1.0-beta
Date Wed, 30 Oct 2013 00:45:11 GMT
I am also working on porting the trunk code onto 2.2.0. Seems quite many API changes but many
of them are just a rename work.
While Yarn 2.1.0 beta also add some client API for easy interaction with YARN framework, but
there are not many examples on how to use them ( API and wiki doc are both old and not reflecting
the new API), some part of SPARK YARN code will need to be rewritten with the new client API
And I am not quite familiar with the user certification part of code, it might take times
for it seems to me this part of codes also change a little bit, some methods gone, and I don't
find the replacement or they are not need anymore.

Best Regards,
Raymond Liu

From: Matei Zaharia [] 
Sent: Wednesday, October 30, 2013 2:35 AM
Subject: Re: spark-0.8.0 and hadoop-2.1.0-beta

I'm curious, Viren, do you have a patch you could post to build this against YARN 2.1 / 2.2?
It would be nice to see how big the changes are.


On Sep 30, 2013, at 10:14 AM, viren kumar <> wrote:

I was able to get Spark 0.8.0 to compile with Hadoop/Yarn 2.1.0-beta, by following some of
the changes described here:
That should help you build most of it. One change not covered there is the change from ProtoUtils.convertFromProtoFormat(containerToken,
cmAddress) to ConverterUtils.convertFromYarn(containerToken, cmAddress).
Not 100% sure that my changes are correct. 
Hope that helps,

On Sun, Sep 29, 2013 at 8:59 AM, Matei Zaharia <> wrote:
Hi Terence,

YARN's API changed in an incompatible way in Hadoop 2.1.0, so I'd suggest sticking with 2.0.x
for now. We may create a different branch for this version. Unfortunately due to the API change
it may not be possible to support this version while also supporting other widely-used versions
like 0.23.x.


On Sep 29, 2013, at 11:00 AM, Terance Dias <> wrote:

> Hi, I'm trying to build spark-0.8.0 with hadoop-2.1.0-beta.
> I have changed the following properties in SparkBuild.scala file.
> val DEFAULT_HADOOP_VERSION = "2.1.0-beta"
> val DEFAULT_YARN = true
> when i do sbt clean compile, I get an error saying
> [error] /usr/local/spark-0.8.0-incubating/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:42:
not found: type AMRMProtocol
> [error]   private var resourceManager: AMRMProtocol = null
> Thanks,
> Terance.

View raw message