spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <>
Subject Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark
Date Thu, 06 Mar 2014 17:58:31 GMT
On Wed, Feb 26, 2014 at 09:22AM, Sean Owen wrote:
> Side point -- "provides" scope is not the same as an exclude.
> "provides" means, this artifact is used directly by this code (compile
> time), but it is not necessary to package it, since it will be
> available from a runtime container. Exclusions make an artifact, that
> would otherwise be available, unavailable at both compile time and
> run-time.

When used at the assemble phase, exclude doesn't affect compile-time deps.
That's exactly what I was looking for while making Spark build suitable for
Bigtop needs.

The main reason was and is: Bigtop stack guarantees the presence of all
upstream libs and their compatibility. So, redistribution in any form is
potentially harmful and useless, at least.


> SBT appears to have syntax for both, just like Maven. Surely these
> have the same meanings in SBT, and excluding artifacts is accomplished
> with exclude and excludeAll, as seen in the Spark build?
> The assembly and shader stuff in Maven is more about controlling
> exactly how it's put together into an artifact, at the level of files
> even, to stick a license file in or exclude some data file cruft or
> rename dependencies.
> exclusions and shading are necessary evils to be used as sparingly as
> possible. Dependency graphs get nuts fast here, and Spark is already
> quite big. (Hence my recent PR to start touching it up -- more coming
> for sure.)
> On Tue, Feb 25, 2014 at 11:20 PM, Evan Chan <> wrote:
> > The correct way to exclude dependencies in SBT is actually to declare
> > a dependency as "provided".   I'm not familiar with Maven or its
> > dependencySet, but provided will mark the entire dependency tree as
> > excluded.   It is also possible to exclude jar by jar, but this is
> > pretty error prone and messy.

View raw message