spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: fixed hbase version in SparkBuild (spark-0.8)
Date Wed, 31 Jul 2013 04:57:21 GMT
Basically the way to think of the assembly is that it should have libraries that users' client
programs need to run. These are core, repl (needed if they use the shell), and likely bagel
and streaming and mllib, though originally we'd opted to leave those out. We are still deciding
on that -- could go either way.


On Jul 30, 2013, at 9:56 PM, Matei Zaharia <> wrote:

> Yeah, that is true. But the assembly shouldn't include the examples project at all IMO
-- if it does now, we should remove it.
> Matei
> On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <> wrote:
>> Matei,
>> Hbase dependencies aren't actually included into the Maven assembly as of this
>> moment, because scope of hbase dependency in examples' module is "compile"; but
>> the assembly is only includes those with "runtime". Hence it is automatically
>> excluded.
>> I believe, hbase is needed for examples during the execution time, and if so -
>> it would have to be fixed in the module. This will lead to need to exclude it
>> from the assembly, in turn.
>> And of course... :)
>>   s/putt/pull/
>> Cos
>> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
>>> Yeah, and maybe we will want to change to Maven as the recommended tool for
>>> assembly building. I want to look into this more for the 0.8 release.
>>> Matei
>>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <> wrote:
>>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
>>>>> Let's at the very least make it configurable, but an even better thing
>>>>> be to make sbt assembly not include it. I think the only thing that depends
>>>>> on HBase is the examples project, but unfortunately SBT puts all its
JARs in
>>>>> the lib_managed folder and just stupidly creates an assembly by grouping
>>>>> those. The Maven build, for example, should not do that.
>>>> It is very easy to exclude dependencies in Maven assembly, like it is done
>>>> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
>>>> you!
>>>> Cos
>>>>> Matei
>>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <>
>>>>>> Hello,
>>>>>> after couple of days(!) of trying to understand where i get the
>>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
>>>>>> hbase.
>>>>>> While it is assumed that hadoop version is specified, hbase version
>>>>>> fixed. This seem to create problem if hbase is used with a particular
>>>>>> version of CDH hadoop client in the backend. (there's a known compatibility
>>>>>> bug).
>>>>>> wouldn't it make sense in this case to allow to declare hbase version
>>>>>> well, perhaps even tie it to the CDH version?
>>>>>> At the very least i think it deserves a specific mention in the header
>>>>>> section to provide opportunity to override, just like hadoop version
>>>>>> Thanks.
>>>>>> -D

View raw message