spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Failed to build Spark with YARN 2.2.0
Date Thu, 24 Oct 2013 17:28:51 GMT
Yup, unfortunately YARN changed its API upon releasing 2.2, which puts us in an awkward position
because all the major current users are on the old YARN API (from 0.23.x and 2.0.x) but new
users will try this one. We'll probably change the default version in Spark 0.8.1 or 0.8.2.
If you look on this list, there were some earlier messages on how to patch Spark to compile
against the new YARN.

Matei

On Oct 23, 2013, at 11:39 PM, Pei-Lun Lee <pllee@appier.com> wrote:

> Hi,
> 
> YARN has already reached stable release 2.2.0. But due to API change Spark cannot build.
> 
> Here is the build log:
> 
> root@master:~/spark# SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true ./sbt/sbt assembly
> [info] Loading project definition from /root/spark/project/project
> [info] Loading project definition from /root/spark/project
> [info] Set current project to root (in build file:/root/spark/)
> [info] Compiling 8 Scala sources to /root/spark/yarn/target/scala-2.9.3/classes...
> [error] /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:42:
not found: type AMRMProtocol
> [error]   private var resourceManager: AMRMProtocol = null
> [error]                                ^
> [error] /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:126:
not found: type AMRMProtocol
> [error]   private def registerWithResourceManager(): AMRMProtocol = {
> [error]                                              ^
> [error] /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:119:
value AM_CONTAINER_ID_ENV is not a member of object org.apache.hadoop.yarn.api.ApplicationConstants
> [error]     val containerIdString = envs.get(ApplicationConstants.AM_CONTAINER_ID_ENV)
> [error]                                                           ^
> [error] /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:131:
not found: type AMRMProtocol
> [error]     return rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> [error]                                 ^
> [error] /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:131:
not found: type AMRMProtocol
> [error]     return rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> [error]                                                                             
^
> [error] /root/spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:138:
value setApplicationAttemptId is not a member of org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest
> [error]     appMasterRequest.setApplicationAttemptId(appAttemptId)
> [error]                      ^
> 
> ... (CUT)
> 
> Is there plan to support YARN 2.2.0?
> 
> Cheers,
> --
> Pei-Lun Lee


Mime
View raw message