spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: fixed hbase version in SparkBuild (spark-0.8)
Date Wed, 31 Jul 2013 04:47:46 GMT
Matei,

Hbase dependencies aren't actually included into the Maven assembly as of this
moment, because scope of hbase dependency in examples' module is "compile"; but
the assembly is only includes those with "runtime". Hence it is automatically
excluded.

I believe, hbase is needed for examples during the execution time, and if so -
it would have to be fixed in the module. This will lead to need to exclude it
from the assembly, in turn.

And of course... :)

    s/putt/pull/

Cos

On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> Yeah, and maybe we will want to change to Maven as the recommended tool for
> assembly building. I want to look into this more for the 0.8 release.
> 
> Matei
> 
> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> >> Let's at the very least make it configurable, but an even better thing will
> >> be to make sbt assembly not include it. I think the only thing that depends
> >> on HBase is the examples project, but unfortunately SBT puts all its JARs in
> >> the lib_managed folder and just stupidly creates an assembly by grouping
> >> those. The Maven build, for example, should not do that.
> > 
> > It is very easy to exclude dependencies in Maven assembly, like it is done for
> > Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
> > you!
> > 
> > Cos
> > 
> >> Matei
> >> 
> >> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:
> >> 
> >>> Hello,
> >>> 
> >>> after couple of days(!) of trying to understand where i get the
> >>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
> >>> hbase.
> >>> 
> >>> While it is assumed that hadoop version is specified, hbase version is
> >>> fixed. This seem to create problem if hbase is used with a particular
> >>> version of CDH hadoop client in the backend. (there's a known compatibility
> >>> bug).
> >>> 
> >>> wouldn't it make sense in this case to allow to declare hbase version as
> >>> well, perhaps even tie it to the CDH version?
> >>> 
> >>> At the very least i think it deserves a specific mention in the header
> >>> section to provide opportunity to override, just like hadoop version does?
> >>> 
> >>> Thanks.
> >>> -D
> >> 
> 

Mime
View raw message