spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject Re: Discussion: Consolidating Spark's build system
Date Thu, 18 Jul 2013 17:56:18 GMT
There is also an alternative called 'sbt-onejar" we can look at.


On Thu, Jul 18, 2013 at 10:48 AM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> Thanks for the feedback. It looks like there are more advantages to Maven
> than I was originally thinking of -- specifically, the better dependency
> resolution and assembly construction. (SBT assembly just takes all the JARs
> in lib_managed and packages them together unfortunately, which means you
> sometimes get multiple versions of the same artifact if you aren't very
> careful with exclusion rules). I think what we'll do is to wait until we
> see whether we can have a single Spark artifact that works with any Hadoop
> version, and go back to the build system issue then.
>
> Matei
>
> On Jul 16, 2013, at 4:37 PM, Ryan LeCompte <lecompte@gmail.com> wrote:
>
> +1 for SBT. It's pretty much the de facto standard for mainstream Scala
> projects.
>
>
> On Tue, Jul 16, 2013 at 4:19 PM, Evan Chan <ev@ooyala.com> wrote:
>
>> If the APIs for those libraries such as Akka stay the same, you don't
>> need different branches.   In SBT you can easily support two different sets
>> of deps depending on which Scala version you are building.... not sure if
>> you could do that with Maven.
>>
>> Here are the main differences that would require a different branch:
>> - if you are taking advantage of Akka Cluster, or features exclusive to
>> newer releases
>> - The Akka futures API changed packages between Scala 2.9.1/2 and Scala
>> 2.9.3/2.10.  However, since the project upgraded to Scala 2.9.3, it should
>> migrate all use of futures to the scala.concurrent.* namespace to avoid
>> more code changes down the line.
>>
>> As far as SBT vs Maven vs Gradle etc.....
>> - I personally think SBT's "console" REPL with all dependencies in the
>> classpath is super valuable for development, ad hoc testing, trying new
>> ideas out.  Not sure if Maven has that.
>> - SBT also has triggered recompilation/testing, which is again very
>> valuable for some of us
>> - Gradle has a easier to understand DSL, but it seems once you build a
>> complex enough system, it is no easier to maintain than SBT
>> - Maven is much better integrated into most IDEs, but SBT is far superior
>> for dev workflow if you don't use one of the major IDEs or are a command
>> line person.
>>
>> -Evan
>>
>>
>> On Tuesday, July 16, 2013 1:37:41 PM UTC-7, Matei Zaharia wrote:
>>
>>> Unfortunately, we'll probably have to have different branches of Spark
>>> for different Scala versions, because there are also other libraries we
>>> depend on (e.g. Akka) that have separate versions for Scala 2.10. You can
>>> actually find a Scala 2.10 port of Spark in the scala-2.10 branch on
>>> GitHub.
>>>
>>> Matei
>>>
>>>
>>> On Jul 16, 2013, at 1:33 PM, Cody Koeninger <co...@koeninger.org>
>>> wrote:
>>>
>>> > If you're looking at consolidating build systems, I'd ask to consider
>>> ease of cross-publishing for different Scala versions.  My instinct is that
>>> sbt will be less troublesome in that regard (although as I understand it,
>>> the changes to the repl may present a problem).
>>> >
>>> > We're needing to use 2.10 for a project, so I'd be happy to put in
>>> some work on the issue.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Spark Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to spark-developers+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-developers+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Spark Developers" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/spark-developers/OxL268v0-Qs/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> spark-developers+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>



-- 
--
Evan Chan
Staff Engineer
ev@ooyala.com  |

<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message