spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Costa <igorco...@apache.org>
Subject Re: Ivy support in Spark vs. sbt
Date Mon, 08 Jun 2015 02:44:59 GMT
Marcelo

I've run this problem once, when I was starting with Spark, like you
mentioned. I found out that ivy get messy with diff sbt version.

My solution was using a previous compatible version with sbt to not cross
with version.


Best
Igor

On Thu, Jun 4, 2015 at 8:08 PM, Eron Wright <ewright@live.com> wrote:

> I saw something like this last night, with a similar message.  Is this
> what you’re referring to?
>
> [error]
> org.deeplearning4j#dl4j-spark-ml;0.0.3.3.4.alpha1-SNAPSHOT!dl4j-spark-ml.jar
> origin location must be absolute:
> file:/Users/eron/.m2/repository/org/deeplearning4j/dl4j-spark-ml/0.0.3.3.4.alpha1-SNAPSHOT/dl4j-spark-ml-0.0.3.3.4.alpha1-SNAPSHOT.jar
> java.lang.IllegalArgumentException:
> org.deeplearning4j#dl4j-spark-ml;0.0.3.3.4.alpha1-SNAPSHOT!dl4j-spark-ml.jar
> origin location must be absolute:
> file:/Users/eron/.m2/repository/org/deeplearning4j/dl4j-spark-ml/0.0.3.3.4.alpha1-SNAPSHOT/dl4j-spark-ml-0.0.3.3.4.alpha1-SNAPSHOT.jar
>         at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:57)
>         at
> org.apache.ivy.core.cache.DefaultRepositoryCacheManager.getArchiveFileInCache(DefaultRepositoryCacheManager.java:385)
>         at
> org.apache.ivy.core.cache.DefaultRepositoryCacheManager.download(DefaultRepositoryCacheManager.java:849)
>         at
> org.apache.ivy.plugins.resolver.BasicResolver.download(BasicResolver.java:835)
>         at
> org.apache.ivy.plugins.resolver.RepositoryResolver.download(RepositoryResolver.java:282)
>         at
> org.apache.ivy.plugins.resolver.ChainResolver.download(ChainResolver.java:219)
>         at
> org.apache.ivy.plugins.resolver.ChainResolver.download(ChainResolver.java:219)
>         at
> org.apache.ivy.core.resolve.ResolveEngine.downloadArtifacts(ResolveEngine.java:388)
>         at
> org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:331)
>         at org.apache.ivy.Ivy.resolve(Ivy.java:517)
>         at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:266)
>         at
> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:175)
>         at
> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:157)
>         at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:151)
>         at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:151)
>         at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:128)
>         at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
>         at sbt.IvySbt$$anon$4.call(Ivy.scala:64)
>         at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
>         at
> xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
>         at
> xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
>         at xsbt.boot.Using$.withResource(Using.scala:10)
>         at xsbt.boot.Using$.apply(Using.scala:9)
>         at
> xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
>         at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
>         at xsbt.boot.Locks$.apply0(Locks.scala:31)
>         at xsbt.boot.Locks$.apply(Locks.scala:28)
>         at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
>         at sbt.IvySbt.withIvy(Ivy.scala:123)
>         at sbt.IvySbt.withIvy(Ivy.scala:120)
>         at sbt.IvySbt$Module.withModule(Ivy.scala:151)
>         at sbt.IvyActions$.updateEither(IvyActions.scala:157)
>         at
> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1318)
>         at
> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1315)
>         at
> sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$85.apply(Defaults.scala:1345)
>         at
> sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$85.apply(Defaults.scala:1343)
>         at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
>         at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1348)
>         at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1342)
>         at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
>         at sbt.Classpaths$.cachedUpdate(Defaults.scala:1360)
>         at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1300)
>         at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1275)
>         at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
>         at
> sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
>         at sbt.std.Transform$$anon$4.work(System.scala:63)
>         at
> sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
>         at
> sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
>         at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
>         at sbt.Execute.work(Execute.scala:235)
>         at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
>         at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
>         at
> sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
>         at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> On 6/4/15, 4:29 AM, "Sean Owen" <sowen@cloudera.com> wrote:
>
> >I've definitely seen the "dependency path must be relative" problem,
> >and fixed it by deleting the ivy cache, but I don't know more than
> >this.
> >
> >On Thu, Jun 4, 2015 at 1:33 AM, Marcelo Vanzin <vanzin@cloudera.com>
> wrote:
> >> Hey all,
> >>
> >> I've been bit by something really weird lately and I'm starting to think
> >> it's related to the ivy support we have in Spark, and running unit tests
> >> that use that code.
> >>
> >> The first thing that happens is that after running unit tests,
> sometimes my
> >> sbt builds start failing with error saying something about "dependency
> path
> >> must be relative" (sorry, don't have the exact error around). The
> dependency
> >> path it prints is a "file:" URL.
> >>
> >> I have a feeling that this is because Spark uses Ivy 2.4 while sbt uses
> Ivy
> >> 2.3, and those might be incompatible. So if they get mixed up, things
> can
> >> break.
> >>
> >> The second is that sometimes unit tests fail with some weird error
> >> downloading dependencies. When checking the ivy metadata in
> ~/.ivy2/cache,
> >> the offending dependencies are pointing to my local maven repo (I have
> >> "maven-local" as one of the entries in my ~/.sbt/repositories).
> >>
> >> My feeling in this case is that Spark's version of Ivy somehow doesn't
> >> handle that case.
> >>
> >> So, long story short:
> >>
> >> - Has anyone run into either of these problems?
> >> - Is it possible to set some env variable or something during tests to
> force
> >> them to use their own directory instead of messing up and breaking my
> >> ~/.ivy2?
> >>
> >>
> >> --
> >> Marcelo
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >For additional commands, e-mail: dev-help@spark.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message