tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris K Wensel <ch...@wensel.net>
Subject Re: Guava deps
Date Fri, 27 Feb 2015 17:41:47 GMT
It looks like this is coming down to Ivy vs Maven conflict management.

Maven chooses nearest, Ivy chooses latest.

I'm forcing to guava 11 to be consistent with how maven resolves dependencies.

ckw

> On Feb 26, 2015, at 4:01 PM, Siddharth Seth <sseth@apache.org> wrote:
> 
> Chris,
> 
> This is what I see by running a mvn dependency:tree on hadoop. Both seem to
> be pulling in guava-11. This is a little strange since curator changed to
> 16 in March 2014, and before that were on 14.
> 
> Hadoop-2.6
> [INFO] +- org.apache.curator:curator-framework:jar:2.6.0:compile
> [INFO] |  +- org.apache.curator:curator-client:jar:2.6.0:compile
> [INFO] |  \- com.google.guava:guava:jar:11.0.2:compile
> [INFO] |     \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
> 
> branch-2
> [INFO] +- org.apache.curator:curator-framework:jar:2.7.1:compile
> [INFO] |  +- org.apache.curator:curator-client:jar:2.7.1:compile
> [INFO] |  \- com.google.guava:guava:jar:11.0.2:compile
> [INFO] |     \- com.google.code.findbugs:jsr305:jar:3.0.0:compile (version
> managed from 1.3.9)
> 
> 
> I'm guessing that maven does something here - either excluding the curator
> guava dependency from being pulled in by Hadoop, or it's fixing versions.
> 
> The dist directory for Hadoop contains only the guava-11 jar. The same
> applies to Tez.
> 
> I don't think auth will be using any dependencies from 16 given what the
> maven dependency tree gives.
> There's been a fair amount of discussion on Hadoop jiras to update the
> Guava version - I'm not sure what the resolution is though.
> 
> From the way Hadoop gets only guava11 in it's classpath, I'd say it's safe
> to remove the Guava16 jar in your setup.
> 
> Is first in classpath a valid assumption ? It does work - but I'm not sure
> there's any guarantees / contracts for this to actually work.
> 
> On a related note, Tez may end up shading the guava version it uses - and
> remove it from dependencies that are exported.
> 
> 
> On Thu, Feb 26, 2015 at 10:59 AM, Chris K Wensel <chris@wensel.net> wrote:
> 
>> I’m running Tez 0.6.0 with the default dependency of Hadoop 2.6 and am
>> receiving a fair number of these
>> 
>> Exception in thread
>> "org.apache.hadoop.util.JvmPauseMonitor$Monitor@74173125"
>> java.lang.NoSuchMethodError:
>> com.google.common.base.Stopwatch.elapsedMillis()J
>>        at
>> org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:184)
>>        at java.lang.Thread.run(Thread.java:722)
>> 
>> of which result in a complete failure to run Tez
>> 
>> This is due to Tez/Yarn building against Guava 11.0.2,
>> 
>> org.apache.tez:tez-api:0.6.0
>> |    +--- log4j:log4j:1.2.17
>> |    +--- com.google.guava:guava:11.0.2 -> 16.0.1
>> 
>> but Curator requiring 16.0.1 and forcing the up version
>> 
>> org.apache.curator:curator-framework:2.6.0
>> |    |    |    |    |    +--- org.apache.curator:curator-client:2.6.0
>> |    |    |    |    |    |    +--- org.slf4j:slf4j-api:1.7.6
>> |    |    |    |    |    |    +--- org.apache.zookeeper:zookeeper:3.4.6 (*)
>> |    |    |    |    |    |    \--- com.google.guava:guava:16.0.1
>> 
>> this makes me wonder how Tez 0.6.0 passes any tests (a side effect of
>> maven?), let alone actually run on a cluster as curator is used by auth
>> which is used by common
>> 
>> or is it a happy coincidence that auth doesn’t use any guava api’s not
>> available in 11.0.2? thus it’s safe to force a downgrade (or put 11.0.2
>> first in the classpath).
>> 
>> ckw
>> 
>> —
>> Chris K Wensel
>> chris@wensel.net
>> 
>> 
>> 
>> 
>> 

—
Chris K Wensel
chris@wensel.net





Mime
View raw message