spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Ivy support in Spark vs. sbt
Date Thu, 04 Jun 2015 20:30:12 GMT
Here's one of the types of exceptions I get (this one when running
VersionsSuite from sql/hive):

[info] - 13: create client *** FAILED *** (1 second, 946 milliseconds)
[info]   java.lang.RuntimeException: [download failed:
org.apache.httpcomponents#httpclient;4.2.5!httpclient.jar, download failed:
commons-codec#commons-codec;1.4!commons-codec.jar]
[info]   at
org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:978)

This is the content of the ivy metadata file for that component:

#ivy cached data file for org.apache.httpcomponents#httpclient;4.2.5
#Thu Jun 04 13:26:10 PDT 2015
artifact\:ivy\#ivy\#xml\#1855381640.is-local=true
artifact\:ivy\#ivy\#xml\#1855381640.location=file\:/home/vanzin/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.pom
artifact\:ivy\#ivy\#xml\#1855381640.exists=true
resolver=local-m2-cache
artifact\:httpclient\#pom.original\#pom\#-365933676.original=artifact\:httpclient\#pom.original\#pom\#-365933676
artifact\:ivy\#ivy\#xml\#1855381640.original=artifact\:httpclient\#pom.original\#pom\#-365933676
artifact.resolver=local-m2-cache
artifact\:httpclient\#pom.original\#pom\#-365933676.is-local=true
artifact\:httpclient\#pom.original\#pom\#-365933676.location=file\:/home/vanzin/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.pom
artifact\:httpclient\#pom.original\#pom\#-365933676.exists=true


If I delete that file *and* the maven copy of those artifacts, then the
tests pass. But that's really annoying, since I have to use sbt and maven
for different things and I really like the fact that sbt can read the maven
cache directly.


On Thu, Jun 4, 2015 at 10:23 AM, shane knapp <sknapp@berkeley.edu> wrote:

> interesting...  i definitely haven't seen it happen that often in our
> build system, and when it has happened, i wasn't able to determine the
> cause.
>
> On Thu, Jun 4, 2015 at 10:16 AM, Marcelo Vanzin <vanzin@cloudera.com>
> wrote:
>
>> On Thu, Jun 4, 2015 at 10:04 AM, shane knapp <sknapp@berkeley.edu> wrote:
>>
>>> this has occasionally happened on our jenkins as well (twice since last
>>> august), and deleting the cache fixes it right up.
>>>
>>
>> Yes deleting the cache fixes things, but it's kinda annoying to have to
>> do that. And yesterday when I was testing a patch that actually used the
>> ivy feature, I had to do that multiple times... that slows things down a
>> lot.
>>
>>
>>>
>>> On Thu, Jun 4, 2015 at 4:29 AM, Sean Owen <sowen@cloudera.com> wrote:
>>>
>>>> I've definitely seen the "dependency path must be relative" problem,
>>>> and fixed it by deleting the ivy cache, but I don't know more than
>>>> this.
>>>>
>>>> On Thu, Jun 4, 2015 at 1:33 AM, Marcelo Vanzin <vanzin@cloudera.com>
>>>> wrote:
>>>> > Hey all,
>>>> >
>>>> > I've been bit by something really weird lately and I'm starting to
>>>> think
>>>> > it's related to the ivy support we have in Spark, and running unit
>>>> tests
>>>> > that use that code.
>>>> >
>>>> > The first thing that happens is that after running unit tests,
>>>> sometimes my
>>>> > sbt builds start failing with error saying something about
>>>> "dependency path
>>>> > must be relative" (sorry, don't have the exact error around). The
>>>> dependency
>>>> > path it prints is a "file:" URL.
>>>> >
>>>> > I have a feeling that this is because Spark uses Ivy 2.4 while sbt
>>>> uses Ivy
>>>> > 2.3, and those might be incompatible. So if they get mixed up, things
>>>> can
>>>> > break.
>>>> >
>>>> > The second is that sometimes unit tests fail with some weird error
>>>> > downloading dependencies. When checking the ivy metadata in
>>>> ~/.ivy2/cache,
>>>> > the offending dependencies are pointing to my local maven repo (I have
>>>> > "maven-local" as one of the entries in my ~/.sbt/repositories).
>>>> >
>>>> > My feeling in this case is that Spark's version of Ivy somehow doesn't
>>>> > handle that case.
>>>> >
>>>> > So, long story short:
>>>> >
>>>> > - Has anyone run into either of these problems?
>>>> > - Is it possible to set some env variable or something during tests
>>>> to force
>>>> > them to use their own directory instead of messing up and breaking my
>>>> > ~/.ivy2?
>>>> >
>>>> >
>>>> > --
>>>> > Marcelo
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>
>>>>
>>>
>>
>>
>> --
>> Marcelo
>>
>
>


-- 
Marcelo

Mime
View raw message