spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: fixed hbase version in SparkBuild (spark-0.8)
Date Wed, 31 Jul 2013 04:56:03 GMT
Yeah, that is true. But the assembly shouldn't include the examples project at all IMO -- if
it does now, we should remove it.


On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <> wrote:

> Matei,
> Hbase dependencies aren't actually included into the Maven assembly as of this
> moment, because scope of hbase dependency in examples' module is "compile"; but
> the assembly is only includes those with "runtime". Hence it is automatically
> excluded.
> I believe, hbase is needed for examples during the execution time, and if so -
> it would have to be fixed in the module. This will lead to need to exclude it
> from the assembly, in turn.
> And of course... :)
>    s/putt/pull/
> Cos
> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
>> Yeah, and maybe we will want to change to Maven as the recommended tool for
>> assembly building. I want to look into this more for the 0.8 release.
>> Matei
>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <> wrote:
>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
>>>> Let's at the very least make it configurable, but an even better thing will
>>>> be to make sbt assembly not include it. I think the only thing that depends
>>>> on HBase is the examples project, but unfortunately SBT puts all its JARs
>>>> the lib_managed folder and just stupidly creates an assembly by grouping
>>>> those. The Maven build, for example, should not do that.
>>> It is very easy to exclude dependencies in Maven assembly, like it is done for
>>> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
>>> you!
>>> Cos
>>>> Matei
>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <> wrote:
>>>>> Hello,
>>>>> after couple of days(!) of trying to understand where i get the
>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
>>>>> hbase.
>>>>> While it is assumed that hadoop version is specified, hbase version is
>>>>> fixed. This seem to create problem if hbase is used with a particular
>>>>> version of CDH hadoop client in the backend. (there's a known compatibility
>>>>> bug).
>>>>> wouldn't it make sense in this case to allow to declare hbase version
>>>>> well, perhaps even tie it to the CDH version?
>>>>> At the very least i think it deserves a specific mention in the header
>>>>> section to provide opportunity to override, just like hadoop version
>>>>> Thanks.
>>>>> -D

View raw message