spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arush Kharbanda <ar...@sigmoidanalytics.com>
Subject Re: Dependency unresolved hadoop-yarn-common 1.0.4 when running quickstart example
Date Thu, 29 Jan 2015 09:33:22 GMT
Hi Sarwar,

For a quick fix you can exclude dependencies for yarn(you wont be needing
them if you are running locally).

libraryDependencies +=
  "log4j" % "log4j" % "1.2.15" exclude("javax.jms", "jms")


You can also analyze your dependencies using this plugin

https://github.com/jrudolph/sbt-dependency-graph


Thanks
Arush

On Thu, Jan 29, 2015 at 4:20 AM, Sarwar Bhuiyan <sarwar.bhuiyan@gmail.com>
wrote:

> Hello all,
>
> I'm trying to build the sample application on the spark 1.2.0 quickstart
> page (https://spark.apache.org/docs/latest/quick-start.html) using the
> following build.sbt file:
>
> name := "Simple Project"
>
> version := "1.0"
>
> scalaVersion := "2.10.4"
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
>
> Upon calling sbt package, it downloaded a lot of dependencies but
> eventually failed with some warnings and errors. Here's the snippet:
>
> [warn]  ::::::::::::::::::::::::::::::::::::::::::::::
> [warn]  ::          UNRESOLVED DEPENDENCIES         ::
> [warn]  ::::::::::::::::::::::::::::::::::::::::::::::
> [warn]  :: org.apache.hadoop#hadoop-yarn-common;1.0.4: not found
> [warn]  :: org.apache.hadoop#hadoop-yarn-client;1.0.4: not found
> [warn]  :: org.apache.hadoop#hadoop-yarn-api;1.0.4: not found
> [warn]  ::::::::::::::::::::::::::::::::::::::::::::::
> [warn]  ::::::::::::::::::::::::::::::::::::::::::::::
> [warn]  ::              FAILED DOWNLOADS            ::
> [warn]  :: ^ see resolution messages for details  ^ ::
> [warn]  ::::::::::::::::::::::::::::::::::::::::::::::
> [warn]  ::
> org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit
> [warn]  ::
> org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit
> [warn]  ::
> org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit
> [warn]  ::
> org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit
> [warn]  ::::::::::::::::::::::::::::::::::::::::::::::
>
>
>
> Upon checking the maven repositories there doesn't seem to be any
> hadoop-yarn-common 1.0.4. I've tried explicitly setting a dependency to
> hadoop-yarn-common 2.4.0 for example but to no avail. I've also tried
> setting a number of different repositories to see if maybe one of them
> might have that dependency. Still no dice.
>
> What's the best way to resolve this for a quickstart situation? Do I have
> to set some sort of profile or environment variable which doesn't try to
> bring the 1.0.4 yarn version?
>
> Any help would be greatly appreciated.
>
> Sarwar
>
>


-- 

[image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>

*Arush Kharbanda* || Technical Teamlead

arush@sigmoidanalytics.com || www.sigmoidanalytics.com

Mime
View raw message