spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Ivy support in Spark vs. sbt
Date Thu, 04 Jun 2015 11:29:23 GMT
I've definitely seen the "dependency path must be relative" problem,
and fixed it by deleting the ivy cache, but I don't know more than
this.

On Thu, Jun 4, 2015 at 1:33 AM, Marcelo Vanzin <vanzin@cloudera.com> wrote:
> Hey all,
>
> I've been bit by something really weird lately and I'm starting to think
> it's related to the ivy support we have in Spark, and running unit tests
> that use that code.
>
> The first thing that happens is that after running unit tests, sometimes my
> sbt builds start failing with error saying something about "dependency path
> must be relative" (sorry, don't have the exact error around). The dependency
> path it prints is a "file:" URL.
>
> I have a feeling that this is because Spark uses Ivy 2.4 while sbt uses Ivy
> 2.3, and those might be incompatible. So if they get mixed up, things can
> break.
>
> The second is that sometimes unit tests fail with some weird error
> downloading dependencies. When checking the ivy metadata in ~/.ivy2/cache,
> the offending dependencies are pointing to my local maven repo (I have
> "maven-local" as one of the entries in my ~/.sbt/repositories).
>
> My feeling in this case is that Spark's version of Ivy somehow doesn't
> handle that case.
>
> So, long story short:
>
> - Has anyone run into either of these problems?
> - Is it possible to set some env variable or something during tests to force
> them to use their own directory instead of messing up and breaking my
> ~/.ivy2?
>
>
> --
> Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message