spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: fixed hbase version in SparkBuild (spark-0.8)
Date Wed, 31 Jul 2013 05:07:25 GMT
Ok, that makes sense, although I thought examples might be a good part of the
documentation for the beginners' references, etc. But I have problems modifying
the assembly accordingly. Will do it the first thing in the morning.

Thanks for the input.
  Cos

On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> Basically the way to think of the assembly is that it should have libraries
> that users' client programs need to run. These are core, repl (needed if
> they use the shell), and likely bagel and streaming and mllib, though
> originally we'd opted to leave those out. We are still deciding on that --
> could go either way.
> 
> Matei
> 
> On Jul 30, 2013, at 9:56 PM, Matei Zaharia <matei.zaharia@gmail.com> wrote:
> 
> > Yeah, that is true. But the assembly shouldn't include the examples project at all
IMO -- if it does now, we should remove it.
> > 
> > Matei
> > 
> > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <cos@apache.org> wrote:
> > 
> >> Matei,
> >> 
> >> Hbase dependencies aren't actually included into the Maven assembly as of this
> >> moment, because scope of hbase dependency in examples' module is "compile";
but
> >> the assembly is only includes those with "runtime". Hence it is automatically
> >> excluded.
> >> 
> >> I believe, hbase is needed for examples during the execution time, and if so
-
> >> it would have to be fixed in the module. This will lead to need to exclude it
> >> from the assembly, in turn.
> >> 
> >> And of course... :)
> >> 
> >>   s/putt/pull/
> >> 
> >> Cos
> >> 
> >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> >>> Yeah, and maybe we will want to change to Maven as the recommended tool
for
> >>> assembly building. I want to look into this more for the 0.8 release.
> >>> 
> >>> Matei
> >>> 
> >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <cos@apache.org> wrote:
> >>> 
> >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> >>>>> Let's at the very least make it configurable, but an even better
thing will
> >>>>> be to make sbt assembly not include it. I think the only thing that
depends
> >>>>> on HBase is the examples project, but unfortunately SBT puts all
its JARs in
> >>>>> the lib_managed folder and just stupidly creates an assembly by
grouping
> >>>>> those. The Maven build, for example, should not do that.
> >>>> 
> >>>> It is very easy to exclude dependencies in Maven assembly, like it is
done for
> >>>> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy,
thank
> >>>> you!
> >>>> 
> >>>> Cos
> >>>> 
> >>>>> Matei
> >>>>> 
> >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
wrote:
> >>>>> 
> >>>>>> Hello,
> >>>>>> 
> >>>>>> after couple of days(!) of trying to understand where i get
the
> >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8
now includes
> >>>>>> hbase.
> >>>>>> 
> >>>>>> While it is assumed that hadoop version is specified, hbase
version is
> >>>>>> fixed. This seem to create problem if hbase is used with a particular
> >>>>>> version of CDH hadoop client in the backend. (there's a known
compatibility
> >>>>>> bug).
> >>>>>> 
> >>>>>> wouldn't it make sense in this case to allow to declare hbase
version as
> >>>>>> well, perhaps even tie it to the CDH version?
> >>>>>> 
> >>>>>> At the very least i think it deserves a specific mention in
the header
> >>>>>> section to provide opportunity to override, just like hadoop
version does?
> >>>>>> 
> >>>>>> Thanks.
> >>>>>> -D
> >>>>> 
> >>> 
> > 
> 

Mime
View raw message