spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Ivy support in Spark vs. sbt
Date Thu, 04 Jun 2015 16:50:39 GMT
They're my local builds, so I wouldn't be able to send you any links... and
the error is generally from sbt, not the unit tests. But if there's any
info I can collect when I see the error, let me know.

I'll try "spark.jars.ivy". I wonder if we should just set that to the
system properties in Spark's root pom.

On Thu, Jun 4, 2015 at 9:47 AM, Burak Yavuz <brkyvz@gmail.com> wrote:

> Hi Marcelo,
>
> This is interesting. Can you please send me links to any failing builds if
> you see that problem please. For now you can set a conf: `spark.jars.ivy`
> to use a path except `~/.ivy2` for Spark.
>
> Thanks,
> Burak
>
> On Thu, Jun 4, 2015 at 4:29 AM, Sean Owen <sowen@cloudera.com> wrote:
>
>> I've definitely seen the "dependency path must be relative" problem,
>> and fixed it by deleting the ivy cache, but I don't know more than
>> this.
>>
>> On Thu, Jun 4, 2015 at 1:33 AM, Marcelo Vanzin <vanzin@cloudera.com>
>> wrote:
>> > Hey all,
>> >
>> > I've been bit by something really weird lately and I'm starting to think
>> > it's related to the ivy support we have in Spark, and running unit tests
>> > that use that code.
>> >
>> > The first thing that happens is that after running unit tests,
>> sometimes my
>> > sbt builds start failing with error saying something about "dependency
>> path
>> > must be relative" (sorry, don't have the exact error around). The
>> dependency
>> > path it prints is a "file:" URL.
>> >
>> > I have a feeling that this is because Spark uses Ivy 2.4 while sbt uses
>> Ivy
>> > 2.3, and those might be incompatible. So if they get mixed up, things
>> can
>> > break.
>> >
>> > The second is that sometimes unit tests fail with some weird error
>> > downloading dependencies. When checking the ivy metadata in
>> ~/.ivy2/cache,
>> > the offending dependencies are pointing to my local maven repo (I have
>> > "maven-local" as one of the entries in my ~/.sbt/repositories).
>> >
>> > My feeling in this case is that Spark's version of Ivy somehow doesn't
>> > handle that case.
>> >
>> > So, long story short:
>> >
>> > - Has anyone run into either of these problems?
>> > - Is it possible to set some env variable or something during tests to
>> force
>> > them to use their own directory instead of messing up and breaking my
>> > ~/.ivy2?
>> >
>> >
>> > --
>> > Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>


-- 
Marcelo

Mime
View raw message