spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Discussion: Consolidating Spark's build system
Date Thu, 18 Jul 2013 17:48:34 GMT
Thanks for the feedback. It looks like there are more advantages to Maven than I was originally
thinking of -- specifically, the better dependency resolution and assembly construction. (SBT
assembly just takes all the JARs in lib_managed and packages them together unfortunately,
which means you sometimes get multiple versions of the same artifact if you aren't very careful
with exclusion rules). I think what we'll do is to wait until we see whether we can have a
single Spark artifact that works with any Hadoop version, and go back to the build system
issue then.

Matei

On Jul 16, 2013, at 4:37 PM, Ryan LeCompte <lecompte@gmail.com> wrote:

> +1 for SBT. It's pretty much the de facto standard for mainstream Scala projects.
> 
> 
> On Tue, Jul 16, 2013 at 4:19 PM, Evan Chan <ev@ooyala.com> wrote:
> If the APIs for those libraries such as Akka stay the same, you don't need different
branches.   In SBT you can easily support two different sets of deps depending on which Scala
version you are building.... not sure if you could do that with Maven.
> 
> Here are the main differences that would require a different branch:
> - if you are taking advantage of Akka Cluster, or features exclusive to newer releases
> - The Akka futures API changed packages between Scala 2.9.1/2 and Scala 2.9.3/2.10. 
However, since the project upgraded to Scala 2.9.3, it should migrate all use of futures to
the scala.concurrent.* namespace to avoid more code changes down the line.
> 
> As far as SBT vs Maven vs Gradle etc.....  
> - I personally think SBT's "console" REPL with all dependencies in the classpath is super
valuable for development, ad hoc testing, trying new ideas out.  Not sure if Maven has that.
> - SBT also has triggered recompilation/testing, which is again very valuable for some
of us
> - Gradle has a easier to understand DSL, but it seems once you build a complex enough
system, it is no easier to maintain than SBT
> - Maven is much better integrated into most IDEs, but SBT is far superior for dev workflow
if you don't use one of the major IDEs or are a command line person.
> 
> -Evan
> 
> 
> On Tuesday, July 16, 2013 1:37:41 PM UTC-7, Matei Zaharia wrote:
> Unfortunately, we'll probably have to have different branches of Spark for different
Scala versions, because there are also other libraries we depend on (e.g. Akka) that have
separate versions for Scala 2.10. You can actually find a Scala 2.10 port of Spark in the
scala-2.10 branch on GitHub. 
> 
> Matei 
> 
> 
> On Jul 16, 2013, at 1:33 PM, Cody Koeninger <co...@koeninger.org> wrote: 
> 
> > If you're looking at consolidating build systems, I'd ask to consider ease of cross-publishing
for different Scala versions.  My instinct is that sbt will be less troublesome in that regard
(although as I understand it, the changes to the repl may present a problem). 
> > 
> > We're needing to use 2.10 for a project, so I'd be happy to put in some work on
the issue. 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "Spark Developers"
group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spark-developers+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "Spark Developers"
group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spark-developers+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message