spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony May <>
Subject Re: Spark scala development in Sbt vs Maven
Date Mon, 05 Mar 2018 17:04:43 GMT
We use sbt for easy cross project dependencies with multiple scala versions
in a mono-repo for which it pretty good albeit with some quirks. As our
projects have matured and change less we moved away from cross project
dependencies but it was extremely useful early in the projects. We knew
that a lot of this was possible in maven/gradle but did not want to go
through the hackage required to get it working.

On Mon, 5 Mar 2018 at 09:49 Sean Owen <> wrote:

> Spark uses Maven as the primary build, but SBT works as well. It reads the
> Maven build to some extent.
> Zinc incremental compilation works with Maven (with the Scala plugin for
> Maven).
> Myself, I prefer Maven, for some of the reasons it is the main build in
> Spark: declarative builds end up being a good thing. You want builds very
> standard. I think the flexibility of writing code to express your build
> just gives a lot of rope to hang yourself with, and recalls the old days of
> Ant builds, where no two builds you'd encounter looked alike when doing the
> same thing.
> If by cross publishing you mean handling different scala versions, yeah
> SBT is more aware of that. The Spark Maven build manages to handle that
> with some hacking.
> On Mon, Mar 5, 2018 at 9:56 AM Jörn Franke <> wrote:
>> I think most of the scala development in Spark happens with sbt - in the
>> open source world.
>>  However, you can do it with Gradle and Maven as well. It depends on your
>> organization etc. what is your standard.
>> Some things might be more cumbersome too reach in non-sbt scala
>> scenarios, but this is more and more improving.
>> > On 5. Mar 2018, at 16:47, Swapnil Shinde <>
>> wrote:
>> >
>> > Hello
>> >    SBT's incremental compilation was a huge plus to build spark+scala
>> applications in SBT for some time. It seems Maven can also support
>> incremental compilation with Zinc server. Considering that, I am interested
>> to know communities experience -
>> >
>> > 1. Spark documentation says SBT is being used by many contributors for
>> day to day development mainly because of incremental compilation.
>> Considering Maven is supporting incremental compilation through Zinc, do
>> contributors prefer to change from SBT to maven?
>> >
>> > 2. Any issues /learning experiences with Maven + Zinc?
>> >
>> > 3. Any other reasons to use SBT over Maven for scala development.
>> >
>> > I understand SBT has many other advantages over Maven like cross
>> version publishing etc. but incremental compilation is major need for us. I
>> am more interested to know why Spark contributors/committers prefer SBT for
>> day to day development.
>> >
>> > Any help and advice would help us to direct our evaluations in right
>> direction,
>> >
>> > Thanks
>> > Swapnil
>> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message