spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark
Date Wed, 26 Feb 2014 09:22:21 GMT
Side point -- "provides" scope is not the same as an exclude.
"provides" means, this artifact is used directly by this code (compile
time), but it is not necessary to package it, since it will be
available from a runtime container. Exclusions make an artifact, that
would otherwise be available, unavailable at both compile time and

SBT appears to have syntax for both, just like Maven. Surely these
have the same meanings in SBT, and excluding artifacts is accomplished
with exclude and excludeAll, as seen in the Spark build?

The assembly and shader stuff in Maven is more about controlling
exactly how it's put together into an artifact, at the level of files
even, to stick a license file in or exclude some data file cruft or
rename dependencies.

exclusions and shading are necessary evils to be used as sparingly as
possible. Dependency graphs get nuts fast here, and Spark is already
quite big. (Hence my recent PR to start touching it up -- more coming
for sure.)

On Tue, Feb 25, 2014 at 11:20 PM, Evan Chan <> wrote:
> The correct way to exclude dependencies in SBT is actually to declare
> a dependency as "provided".   I'm not familiar with Maven or its
> dependencySet, but provided will mark the entire dependency tree as
> excluded.   It is also possible to exclude jar by jar, but this is
> pretty error prone and messy.

View raw message