spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <>
Subject Re: Assorted project updates (tests, build, etc)
Date Thu, 03 Jul 2014 22:48:46 GMT
Sorry all, I sent the wrong pull request to refer to Prashant's work:

On Thu, Jul 3, 2014 at 1:37 PM, Patrick Wendell <> wrote:
> Just a reminder here - we'll soon be merging a patch that changes the
> SBT build internals significantly. We've tried to make this fully
> backwards compatible, but there may be issues (which we'll resolve as
> they arrive).
> - Patrick
> On Sun, Jun 22, 2014 at 10:27 AM, Mark Hamstra <> wrote:
>> Just a couple of FYI notes: With Zinc and the scala-maven-plugin, repl and
>> incremental builds are also available to those doing day-to-day development
>> using Maven.  As long as you don't have to delve into the extra boilerplate
>> and verbosity of Maven's POMs relative to an SBT build file, there is
>> little day-to-day functional difference between the two -- if anything, I
>> find that Maven supports faster development cycles.
>> On Sun, Jun 22, 2014 at 12:24 AM, Patrick Wendell <>
>> wrote:
>>> Hey All,
>>> 1. The original test infrastructure hosted by the AMPLab has been
>>> fully restored and also expanded with many more executor slots for
>>> tests. Thanks to Matt Massie at the Amplab for helping with this.
>>> 2. We now have a nightly build matrix across different Hadoop
>>> versions. It appears that the Maven build is failing tests with some
>>> of the newer Hadoop versions. If people from the community are
>>> interested, diagnosing and fixing test issues would be welcome patches
>>> (they are all dependency related).
>>> 3. Prashant Sharma has spent a lot of time to make it possible for our
>>> sbt build to read dependencies from Maven. This will save us a huge
>>> amount of headache keeping the builds consistent. I just wanted to
>>> give a heads up to users about this - we should retain compatibility
>>> with features of the sbt build, but if you are e.g. hooking into deep
>>> internals of our build it may affect you. I'm hoping this can be
>>> updated and merged in the next week:
>>> 4. We've moved most of the documentation over to recommending users
>>> build with Maven when creating official packages. This is just to
>>> provide a single "reference build" of Spark since it's the one we test
>>> and package for releases, we make sure all recursive dependencies are
>>> correct, etc. I'd recommend that all downstream packagers use this
>>> build.
>>> For day-to-day development I imagine sbt will remain more popular
>>> (repl, incremental builds, etc). Prashant's work allows us to get the
>>> "best of both worlds" which is great.
>>> - Patrick

View raw message