spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Rosen <joshro...@databricks.com>
Subject Re: Unresolved dep when building project with spark 1.6
Date Mon, 29 Feb 2016 18:30:51 GMT
Have you tried removing the leveldbjni files from your local ivy cache? My
hunch is that this is a problem with some local cache state rather than the
dependency simply being unavailable / not existing (note that the error
message was "origin location must be absolute:[...]", not that the files
couldn't be found).

On Mon, Feb 29, 2016 at 2:19 AM Hao Ren <invkrh@gmail.com> wrote:

> Hi,
>
> I am upgrading my project to spark 1.6.
> It seems that the deps are broken.
>
> Deps used in sbt
>
> val scalaVersion = "2.10"
> val sparkVersion  = "1.6.0"
> val hadoopVersion = "2.7.1"
>
> // Libraries
> val scalaTest = "org.scalatest" %% "scalatest" % "2.2.4" % "test"
> val sparkSql  = "org.apache.spark" %% "spark-sql" % sparkVersion
> val sparkML   = "org.apache.spark" %% "spark-mllib" % sparkVersion
> val hadoopAWS = "org.apache.hadoop" % "hadoop-aws" % hadoopVersion
> val scopt     = "com.github.scopt" %% "scopt" % "3.3.0"
> val jodacvt   = "org.joda" % "joda-convert" % "1.8.1"
>
> Sbt exception:
>
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] ::          UNRESOLVED DEPENDENCIES         ::
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn] :: org.fusesource.leveldbjni#leveldbjni-all;1.8:
> org.fusesource.leveldbjni#leveldbjni-all;1.8!leveldbjni-all.pom(pom.original)
> origin location must be absolute:
> file:/home/invkrh/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.pom
> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
> [warn]
> [warn] Note: Unresolved dependencies path:
> [warn] org.fusesource.leveldbjni:leveldbjni-all:1.8
> [warn]  +- org.apache.spark:spark-network-shuffle_2.10:1.6.0
> [warn]  +- org.apache.spark:spark-core_2.10:1.6.0
> [warn]  +- org.apache.spark:spark-catalyst_2.10:1.6.0
> [warn]  +- org.apache.spark:spark-sql_2.10:1.6.0
> (/home/invkrh/workspace/scala/confucius/botdet/build.sbt#L14)
> [warn]  +- org.apache.spark:spark-mllib_2.10:1.6.0
> (/home/invkrh/workspace/scala/confucius/botdet/build.sbt#L14)
> [warn]  +- fr.leboncoin:botdet_2.10:0.1
> sbt.ResolveException: unresolved dependency:
> org.fusesource.leveldbjni#leveldbjni-all;1.8:
> org.fusesource.leveldbjni#leveldbjni-all;1.8!leveldbjni-all.pom(pom.original)
> origin location must be absolute:
> file:/home/invkrh/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.pom
>
> Thank you.
>
> --
> Hao Ren
>
> Data Engineer @ leboncoin
>
> Paris, France
>

Mime
View raw message