spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Spark scala development in Sbt vs Maven
Date Mon, 05 Mar 2018 16:49:40 GMT
Spark uses Maven as the primary build, but SBT works as well. It reads the
Maven build to some extent.

Zinc incremental compilation works with Maven (with the Scala plugin for

Myself, I prefer Maven, for some of the reasons it is the main build in
Spark: declarative builds end up being a good thing. You want builds very
standard. I think the flexibility of writing code to express your build
just gives a lot of rope to hang yourself with, and recalls the old days of
Ant builds, where no two builds you'd encounter looked alike when doing the
same thing.

If by cross publishing you mean handling different scala versions, yeah SBT
is more aware of that. The Spark Maven build manages to handle that with
some hacking.

On Mon, Mar 5, 2018 at 9:56 AM Jörn Franke <> wrote:

> I think most of the scala development in Spark happens with sbt - in the
> open source world.
>  However, you can do it with Gradle and Maven as well. It depends on your
> organization etc. what is your standard.
> Some things might be more cumbersome too reach in non-sbt scala scenarios,
> but this is more and more improving.
> > On 5. Mar 2018, at 16:47, Swapnil Shinde <>
> wrote:
> >
> > Hello
> >    SBT's incremental compilation was a huge plus to build spark+scala
> applications in SBT for some time. It seems Maven can also support
> incremental compilation with Zinc server. Considering that, I am interested
> to know communities experience -
> >
> > 1. Spark documentation says SBT is being used by many contributors for
> day to day development mainly because of incremental compilation.
> Considering Maven is supporting incremental compilation through Zinc, do
> contributors prefer to change from SBT to maven?
> >
> > 2. Any issues /learning experiences with Maven + Zinc?
> >
> > 3. Any other reasons to use SBT over Maven for scala development.
> >
> > I understand SBT has many other advantages over Maven like cross version
> publishing etc. but incremental compilation is major need for us. I am more
> interested to know why Spark contributors/committers prefer SBT for day to
> day development.
> >
> > Any help and advice would help us to direct our evaluations in right
> direction,
> >
> > Thanks
> > Swapnil
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message