spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <>
Subject Re: Assorted project updates (tests, build, etc)
Date Thu, 03 Jul 2014 20:37:42 GMT
Just a reminder here - we'll soon be merging a patch that changes the
SBT build internals significantly. We've tried to make this fully
backwards compatible, but there may be issues (which we'll resolve as
they arrive).

- Patrick

On Sun, Jun 22, 2014 at 10:27 AM, Mark Hamstra <> wrote:
> Just a couple of FYI notes: With Zinc and the scala-maven-plugin, repl and
> incremental builds are also available to those doing day-to-day development
> using Maven.  As long as you don't have to delve into the extra boilerplate
> and verbosity of Maven's POMs relative to an SBT build file, there is
> little day-to-day functional difference between the two -- if anything, I
> find that Maven supports faster development cycles.
> On Sun, Jun 22, 2014 at 12:24 AM, Patrick Wendell <>
> wrote:
>> Hey All,
>> 1. The original test infrastructure hosted by the AMPLab has been
>> fully restored and also expanded with many more executor slots for
>> tests. Thanks to Matt Massie at the Amplab for helping with this.
>> 2. We now have a nightly build matrix across different Hadoop
>> versions. It appears that the Maven build is failing tests with some
>> of the newer Hadoop versions. If people from the community are
>> interested, diagnosing and fixing test issues would be welcome patches
>> (they are all dependency related).
>> 3. Prashant Sharma has spent a lot of time to make it possible for our
>> sbt build to read dependencies from Maven. This will save us a huge
>> amount of headache keeping the builds consistent. I just wanted to
>> give a heads up to users about this - we should retain compatibility
>> with features of the sbt build, but if you are e.g. hooking into deep
>> internals of our build it may affect you. I'm hoping this can be
>> updated and merged in the next week:
>> 4. We've moved most of the documentation over to recommending users
>> build with Maven when creating official packages. This is just to
>> provide a single "reference build" of Spark since it's the one we test
>> and package for releases, we make sure all recursive dependencies are
>> correct, etc. I'd recommend that all downstream packagers use this
>> build.
>> For day-to-day development I imagine sbt will remain more popular
>> (repl, incremental builds, etc). Prashant's work allows us to get the
>> "best of both worlds" which is great.
>> - Patrick

View raw message