spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <>
Subject Ivy support in Spark vs. sbt
Date Wed, 03 Jun 2015 23:33:42 GMT
Hey all,

I've been bit by something really weird lately and I'm starting to think
it's related to the ivy support we have in Spark, and running unit tests
that use that code.

The first thing that happens is that after running unit tests, sometimes my
sbt builds start failing with error saying something about "dependency path
must be relative" (sorry, don't have the exact error around). The
dependency path it prints is a "file:" URL.

I have a feeling that this is because Spark uses Ivy 2.4 while sbt uses Ivy
2.3, and those might be incompatible. So if they get mixed up, things can

The second is that sometimes unit tests fail with some weird error
downloading dependencies. When checking the ivy metadata in ~/.ivy2/cache,
the offending dependencies are pointing to my local maven repo (I have
"maven-local" as one of the entries in my ~/.sbt/repositories).

My feeling in this case is that Spark's version of Ivy somehow doesn't
handle that case.

So, long story short:

- Has anyone run into either of these problems?
- Is it possible to set some env variable or something during tests to
force them to use their own directory instead of messing up and breaking my


View raw message