crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject refactoring crunch-archetype
Date Mon, 11 Mar 2013 21:01:14 GMT
Hey Matthias,

I cc'd everyone else on here, but since this was your module, I thought it
best to solicit your opinion before refactoring it.

We never managed to get crunch-archetypes working w/hadoop 2.x, which is
apparently deprecating the lib/* trick for including client dependencies in
favor of the -libjars option (see )

The way that I have found to do this in Maven is to use the
copy-dependencies option of the maven-dependency-plugin and include a shell
script in a bin/ directory that knows how to setup the HADOOP_CLASSPATH and
libjars arguments for use with hadoop jar. Although this approach is more
complex than the lib/* trick, it will be able to support hadoop 1.x as well
as hadoop 2.x.

Do you have any objections to me taking this on, and/or any other landmines
I should keep an eye out for?


Director of Data Science
Cloudera <>
Twitter: @josh_wills <>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message