spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: fixed hbase version in SparkBuild (spark-0.8)
Date Wed, 31 Jul 2013 18:38:51 GMT
On Wed, Jul 31, 2013 at 11:24 AM, Konstantin Boudnik <cos@apache.org> wrote:

> Dmitriy,
>
> lib_managed/ is controlled by sbt build - Maven uses ~/.m2/repository
> directly. Hence, earlier Matei point stands on the need to fix it in sbt
> build.
>
> I am not sure why would Cloudera solve the issue in the Spark build? :)
>

I wasn't saying –°loudera  should. I just meant to say i saw other people
stranded out there on that very issue right now while searching for the
cause. This incompatibility bug is not at all obvious to track.

Ok i guess it should be ok for sbt build to declare HBASE_VERSION and use
it.

I also discovered that cdh hbase counterpart does not seem to be published
to any repository spark uses (unlike hadoop client jars). So to resolve, i
also had to add a direct cloudera resolver. it is probably more than you
want to do for spark, but any person using hbase and cdh would probably run
into this surprisingly hard to track issue.


>
> Cos
>
> On Tue, Jul 30, 2013 at 11:13PM, Dmitriy Lyubimov wrote:
> > FWIW it does get landed in lib_managed...
> >
> > i beleive there's a post in CDH group with the same very problem i had ,
> > still unresolved/unadvised by Cloudera.
> >
> >
> > On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
> >
> > > Ok, that makes sense, although I thought examples might be a good part
> of
> > > the
> > > documentation for the beginners' references, etc. But I have problems
> > > modifying
> > > the assembly accordingly. Will do it the first thing in the morning.
> > >
> > > Thanks for the input.
> > >   Cos
> > >
> > > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > > > Basically the way to think of the assembly is that it should have
> > > libraries
> > > > that users' client programs need to run. These are core, repl
> (needed if
> > > > they use the shell), and likely bagel and streaming and mllib, though
> > > > originally we'd opted to leave those out. We are still deciding on
> that
> > > --
> > > > could go either way.
> > > >
> > > > Matei
> > > >
> > > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <matei.zaharia@gmail.com>
> > > wrote:
> > > >
> > > > > Yeah, that is true. But the assembly shouldn't include the examples
> > > project at all IMO -- if it does now, we should remove it.
> > > > >
> > > > > Matei
> > > > >
> > > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <cos@apache.org>
> > > wrote:
> > > > >
> > > > >> Matei,
> > > > >>
> > > > >> Hbase dependencies aren't actually included into the Maven
> assembly
> > > as of this
> > > > >> moment, because scope of hbase dependency in examples' module
is
> > > "compile"; but
> > > > >> the assembly is only includes those with "runtime". Hence it
is
> > > automatically
> > > > >> excluded.
> > > > >>
> > > > >> I believe, hbase is needed for examples during the execution
time,
> > > and if so -
> > > > >> it would have to be fixed in the module. This will lead to need
to
> > > exclude it
> > > > >> from the assembly, in turn.
> > > > >>
> > > > >> And of course... :)
> > > > >>
> > > > >>   s/putt/pull/
> > > > >>
> > > > >> Cos
> > > > >>
> > > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > > > >>> Yeah, and maybe we will want to change to Maven as the
> recommended
> > > tool for
> > > > >>> assembly building. I want to look into this more for the
0.8
> release.
> > > > >>>
> > > > >>> Matei
> > > > >>>
> > > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <cos@apache.org>
> > > wrote:
> > > > >>>
> > > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > > > >>>>> Let's at the very least make it configurable, but
an even
> better
> > > thing will
> > > > >>>>> be to make sbt assembly not include it. I think the
only thing
> > > that depends
> > > > >>>>> on HBase is the examples project, but unfortunately
SBT puts
> all
> > > its JARs in
> > > > >>>>> the lib_managed folder and just stupidly creates
an assembly by
> > > grouping
> > > > >>>>> those. The Maven build, for example, should not do
that.
> > > > >>>>
> > > > >>>> It is very easy to exclude dependencies in Maven assembly,
like
> it
> > > is done for
> > > > >>>> Hadoop. Lemme send out a putt request - a good finding
indeed,
> > > Dmitriy, thank
> > > > >>>> you!
> > > > >>>>
> > > > >>>> Cos
> > > > >>>>
> > > > >>>>> Matei
> > > > >>>>>
> > > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <
> dlieu.7@gmail.com>
> > > wrote:
> > > > >>>>>
> > > > >>>>>> Hello,
> > > > >>>>>>
> > > > >>>>>> after couple of days(!) of trying to understand
where i get
> the
> > > > >>>>>> "NoSuchMethod" error, i traced it down to the
fact that 0.8
> now
> > > includes
> > > > >>>>>> hbase.
> > > > >>>>>>
> > > > >>>>>> While it is assumed that hadoop version is specified,
hbase
> > > version is
> > > > >>>>>> fixed. This seem to create problem if hbase is
used with a
> > > particular
> > > > >>>>>> version of CDH hadoop client in the backend.
(there's a known
> > > compatibility
> > > > >>>>>> bug).
> > > > >>>>>>
> > > > >>>>>> wouldn't it make sense in this case to allow
to declare hbase
> > > version as
> > > > >>>>>> well, perhaps even tie it to the CDH version?
> > > > >>>>>>
> > > > >>>>>> At the very least i think it deserves a specific
mention in
> the
> > > header
> > > > >>>>>> section to provide opportunity to override, just
like hadoop
> > > version does?
> > > > >>>>>>
> > > > >>>>>> Thanks.
> > > > >>>>>> -D
> > > > >>>>>
> > > > >>>
> > > > >
> > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message