spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: fixed hbase version in SparkBuild (spark-0.8)
Date Wed, 31 Jul 2013 18:24:29 GMT
Dmitriy,

lib_managed/ is controlled by sbt build - Maven uses ~/.m2/repository
directly. Hence, earlier Matei point stands on the need to fix it in sbt
build.

I am not sure why would Cloudera solve the issue in the Spark build? :)

Cos

On Tue, Jul 30, 2013 at 11:13PM, Dmitriy Lyubimov wrote:
> FWIW it does get landed in lib_managed...
> 
> i beleive there's a post in CDH group with the same very problem i had ,
> still unresolved/unadvised by Cloudera.
> 
> 
> On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > Ok, that makes sense, although I thought examples might be a good part of
> > the
> > documentation for the beginners' references, etc. But I have problems
> > modifying
> > the assembly accordingly. Will do it the first thing in the morning.
> >
> > Thanks for the input.
> >   Cos
> >
> > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > > Basically the way to think of the assembly is that it should have
> > libraries
> > > that users' client programs need to run. These are core, repl (needed if
> > > they use the shell), and likely bagel and streaming and mllib, though
> > > originally we'd opted to leave those out. We are still deciding on that
> > --
> > > could go either way.
> > >
> > > Matei
> > >
> > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <matei.zaharia@gmail.com>
> > wrote:
> > >
> > > > Yeah, that is true. But the assembly shouldn't include the examples
> > project at all IMO -- if it does now, we should remove it.
> > > >
> > > > Matei
> > > >
> > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <cos@apache.org>
> > wrote:
> > > >
> > > >> Matei,
> > > >>
> > > >> Hbase dependencies aren't actually included into the Maven assembly
> > as of this
> > > >> moment, because scope of hbase dependency in examples' module is
> > "compile"; but
> > > >> the assembly is only includes those with "runtime". Hence it is
> > automatically
> > > >> excluded.
> > > >>
> > > >> I believe, hbase is needed for examples during the execution time,
> > and if so -
> > > >> it would have to be fixed in the module. This will lead to need to
> > exclude it
> > > >> from the assembly, in turn.
> > > >>
> > > >> And of course... :)
> > > >>
> > > >>   s/putt/pull/
> > > >>
> > > >> Cos
> > > >>
> > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > > >>> Yeah, and maybe we will want to change to Maven as the recommended
> > tool for
> > > >>> assembly building. I want to look into this more for the 0.8 release.
> > > >>>
> > > >>> Matei
> > > >>>
> > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <cos@apache.org>
> > wrote:
> > > >>>
> > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > > >>>>> Let's at the very least make it configurable, but an even
better
> > thing will
> > > >>>>> be to make sbt assembly not include it. I think the only
thing
> > that depends
> > > >>>>> on HBase is the examples project, but unfortunately SBT
puts all
> > its JARs in
> > > >>>>> the lib_managed folder and just stupidly creates an assembly
by
> > grouping
> > > >>>>> those. The Maven build, for example, should not do that.
> > > >>>>
> > > >>>> It is very easy to exclude dependencies in Maven assembly,
like it
> > is done for
> > > >>>> Hadoop. Lemme send out a putt request - a good finding indeed,
> > Dmitriy, thank
> > > >>>> you!
> > > >>>>
> > > >>>> Cos
> > > >>>>
> > > >>>>> Matei
> > > >>>>>
> > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> > wrote:
> > > >>>>>
> > > >>>>>> Hello,
> > > >>>>>>
> > > >>>>>> after couple of days(!) of trying to understand where
i get the
> > > >>>>>> "NoSuchMethod" error, i traced it down to the fact
that 0.8 now
> > includes
> > > >>>>>> hbase.
> > > >>>>>>
> > > >>>>>> While it is assumed that hadoop version is specified,
hbase
> > version is
> > > >>>>>> fixed. This seem to create problem if hbase is used
with a
> > particular
> > > >>>>>> version of CDH hadoop client in the backend. (there's
a known
> > compatibility
> > > >>>>>> bug).
> > > >>>>>>
> > > >>>>>> wouldn't it make sense in this case to allow to declare
hbase
> > version as
> > > >>>>>> well, perhaps even tie it to the CDH version?
> > > >>>>>>
> > > >>>>>> At the very least i think it deserves a specific mention
in the
> > header
> > > >>>>>> section to provide opportunity to override, just like
hadoop
> > version does?
> > > >>>>>>
> > > >>>>>> Thanks.
> > > >>>>>> -D
> > > >>>>>
> > > >>>
> > > >
> > >
> >

Mime
View raw message