spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <>
Subject Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark
Date Wed, 26 Feb 2014 15:17:29 GMT
We maintain in house spark build using sbt. We have no problem using sbt
assembly. We did add a few exclude statements for transitive dependencies.

The main enemy of assemblies are jars that include stuff they shouldn't
(kryo comes to mind, I think they include logback?), new versions of jars
that change the provider/artifact without changing the package (asm), and
incompatible new releases (protobuf). These break the transitive resolution
process. I imagine that's true for any build tool.

Besides shading I don't see anything maven can do sbt cannot, and if I
understand it correctly shading is not done currently using the build tool.

Since spark is primarily scala/akka based the main developer base will be
familiar with sbt (I think?). Switching build tool is always painful. I
personally think it is smarter to put this burden on a limited number of
upstream integrators than on the community. However that said I don't think
its a problem for us to maintain an sbt build in-house if spark switched to
The problem is, the complete spark dependency graph is fairly large,
and there are lot of conflicting versions in there.
In particular, when we bump versions of dependencies - making managing
this messy at best.

Now, I have not looked in detail at how maven manages this - it might
just be accidental that we get a decent out-of-the-box assembled
shaded jar (since we dont do anything great to configure it).
With current state of sbt in spark, it definitely is not a good
solution : if we can enhance it (or it already is ?), while keeping
the management of the version/dependency graph manageable, I dont have
any objections to using sbt or maven !
Too many exclude versions, pinned versions, etc would just make things
unmanageable in future.


On Wed, Feb 26, 2014 at 8:56 AM, Evan chan <> wrote:
> Actually you can control exactly how sbt assembly merges or resolves
conflicts.  I believe the default settings however lead to order which
cannot be controlled.
> I do wish for a smarter fat jar plugin.
> -Evan
> To be free is not merely to cast off one's chains, but to live in a way
that respects & enhances the freedom of others. (#NelsonMandela)
>> On Feb 25, 2014, at 6:50 PM, Mridul Muralidharan <>
>>> On Wed, Feb 26, 2014 at 5:31 AM, Patrick Wendell <>
>>> Evan - this is a good thing to bring up. Wrt the shader plug-in -
>>> right now we don't actually use it for bytecode shading - we simply
>>> use it for creating the uber jar with excludes (which sbt supports
>>> just fine via assembly).
>> Not really - as I mentioned initially in this thread, sbt's assembly
>> does not take dependencies into account properly : and can overwrite
>> newer classes with older versions.
>> From an assembly point of view, sbt is not very good : we are yet to
>> try it after 2.10 shift though (and probably wont, given the mess it
>> created last time).
>> Regards,
>> Mridul
>>> I was wondering actually, do you know if it's possible to added shaded
>>> artifacts to the *spark jar* using this plug-in (e.g. not an uber
>>> jar)? That's something I could see being really handy in the future.
>>> - Patrick
>>>> On Tue, Feb 25, 2014 at 3:39 PM, Evan Chan <> wrote:
>>>> The problem is that plugins are not equivalent.  There is AFAIK no
>>>> equivalent to the maven shader plugin for SBT.
>>>> There is an SBT plugin which can apparently read POM XML files
>>>> (sbt-pom-reader).   However, it can't possibly handle plugins, which
>>>> is still problematic.
>>>>> On Tue, Feb 25, 2014 at 3:31 PM, yao <> wrote:
>>>>> I would prefer keep both of them, it would be better even if that
>>>>> pom.xml will be generated using sbt. Some company, like my current
>>>>> have their own build infrastructures built on top of maven. It is not
>>>>> to support sbt for these potential spark clients. But I do agree to
>>>>> keep one if there is a promising way to generate correct
configuration from
>>>>> the other.
>>>>> -Shengzhe
>>>>>> On Tue, Feb 25, 2014 at 3:20 PM, Evan Chan <>
>>>>>> The correct way to exclude dependencies in SBT is actually to declare
>>>>>> a dependency as "provided".   I'm not familiar with Maven or its
>>>>>> dependencySet, but provided will mark the entire dependency tree
>>>>>> excluded.   It is also possible to exclude jar by jar, but this is
>>>>>> pretty error prone and messy.
>>>>>>> On Tue, Feb 25, 2014 at 2:45 PM, Koert Kuipers <>
>>>>>>> yes in sbt assembly you can exclude jars (although i never had
need for
>>>>>>> this) and files in jars.
>>>>>>> for example i frequently remove, because for
>>>>>>> reason hadoop decided to include it making it very difficult
to use
>>>>>> own
>>>>>>> logging config.
>>>>>>>> On Tue, Feb 25, 2014 at 4:24 PM, Konstantin Boudnik <
>>>>>>> wrote:
>>>>>>>>> On Fri, Feb 21, 2014 at 11:11AM, Patrick Wendell wrote:
>>>>>>>>> Kos - thanks for chiming in. Could you be more specific
what is
>>>>>>>>> available in maven and not in sbt for these issues? I
took a look
>>>>>>>>> the bigtop code relating to Spark. As far as I could
tell [1] was
>>>>>>>>> main point of integration with the build system (maybe
there are
>>>>>>>>> integration points)?
>>>>>>>>>>  - in order to integrate Spark well into existing
Hadoop stack it
>>>>>> was
>>>>>>>>>>    necessary to have a way to avoid transitive dependencies
>>>>>>>> duplications and
>>>>>>>>>>    possible conflicts.
>>>>>>>>>>    E.g. Maven assembly allows us to avoid adding
_all_ Hadoop
>>>>>>>> and later
>>>>>>>>>>    merely declare Spark package dependency on standard
>>>>>> Hadoop
>>>>>>>>>>    packages. And yes - Bigtop packaging means the
naming and
>>>>>>>> would be
>>>>>>>>>>    standard across all commercial Hadoop distributions
that are
>>>>>> worth
>>>>>>>>>>    mentioning: ASF Bigtop convenience binary packages,
>>>>>> Cloudera or
>>>>>>>>>>    Hortonworks packages. Hence, the downstream user
doesn't need
>>>>>>>> spend any
>>>>>>>>>>    effort to make sure that Spark "clicks-in" properly.
>>>>>>>>> The sbt build also allows you to plug in a Hadoop version
>>>>>>>>> the maven build.
>>>>>>>> I am actually talking about an ability to exclude a set of
>>>>>>>> from an
>>>>>>>> assembly, similarly to what's happening in dependencySet
>>>>>>>>    assembly/src/main/assembly/assembly.xml
>>>>>>>> If there is a comparable functionality in Sbt, that would
quite a
>>>>>> bit,
>>>>>>>> apparently.
>>>>>>>> Cos
>>>>>>>>>>  - Maven provides a relatively easy way to deal with
the jar-hell
>>>>>>>> problem,
>>>>>>>>>>    although the original maven build was just Shader'ing
>>>>>>>> into a
>>>>>>>>>>    huge lump of class files. Oftentimes ending up
with classes
>>>>>>>> slamming on
>>>>>>>>>>    top of each other from different transitive dependencies.
>>>>>>>>> AFIAK we are only using the shade plug-in to deal with
>>>>>>>>> resolution in the assembly jar. These are dealt with
in sbt via
>>>>>>>>> sbt assembly plug-in in an identical way. Is there a
>>>>>>>> I am bringing up the Sharder, because it is an awful hack,
which is
>>>>>> can't
>>>>>>>> be
>>>>>>>> used in real controlled deployment.
>>>>>>>> Cos
>>>>>>>>> [1]
>>>>>> --
>>>>>> --
>>>>>> Evan Chan
>>>>>> Staff Engineer
>>>>>>  |
>>>> --
>>>> --
>>>> Evan Chan
>>>> Staff Engineer
>>>>  |

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message