spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Suereth <joshua.suer...@gmail.com>
Subject Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark
Date Wed, 26 Mar 2014 23:26:48 GMT
Cool. It sounds like focusing on sbt-pom-reader would be a good thing for
you guys then.

There's a few... fun... issues around maven parent projects that are still
running around with sbt-pom-reader that appear to be fundamental ivy-maven
hate-based issues.

IN any case, while I'm generally swamped with things, let me know anything
I can do to improve the situation.  The shading thing frustrates me
slightly, as I thinka lot of projects that shade have been abandoned (that
I looked into).  Which plugin are you using?


On Fri, Mar 14, 2014 at 10:21 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> I like the pom-reader approach as well -- in particular, that it lets you
> add extra stuff in your SBT build after loading the dependencies from the
> POM. Profiles would be the one thing missing to be able to pass options
> through.
>
> Matei
>
> On Mar 14, 2014, at 10:03 AM, Patrick Wendell <pwendell@gmail.com> wrote:
>
> > Hey Josh,
> >
> > I spent some time playing with the sbt-pom-reader and actually I think
> > it might suite our needs pretty well. Basically our situation is that
> > Maven has more mature support for packaging (shading, official
> > assembly plug-in) and is used by pretty much every downstream packager
> > of Spark. SBT is used by the developers heavily for incremental
> > compilation, the console, etc. So far we've been maintaining entirely
> > isolated builds in sbt and maven which is a huge development cost and
> > something that's lead to divergence over time.
> >
> > The sbt-pom-reader I think could give us the best of both worlds. But
> > it doesn't support profiles in Maven which we use pretty heavily. I
> > played a bit this week and tried to get it to work without much luck:
> >
> https://github.com/pwendell/sbt-pom-reader/commit/ca1f1f2d6bf8891acb7212facf4807baaca8974d
> >
> > Any pointers or interest in helping us with that feature? I think with
> > that in place we might have a good solution where the dependencies are
> > declared in one place (poms) and we use Maven for packaging but we can
> > use sbt for day to day development.
> >
> >
> >
> > On Fri, Mar 14, 2014 at 9:41 AM, Evan Chan <ev@ooyala.com> wrote:
> >> Jsuereth -
> >>
> >> Thanks for jumping on this thread.  There are a couple things which
> >> would help SBT IMHO (w.r.t. issues mentioned here):
> >>
> >> - A way of generating a project-wide pom.xml  (I believe make-pom only
> >> generates it for each sub-project)
> >>   - and probably better or more feature-complete POMs, but the Maven
> >> folks here can speak to that
> >> - Make the sbt-pom-reader plugin officially part of SBT, and make it
> >> work well (again the Maven folks need to jump in here, though plugins
> >> can't be directly translated)
> >> - Have the sbt-assembly plugin officially maintained by Typesafe, or
> >> part of SBT....  most Maven folks expect not to have to include a
> >> plugin to generate a fat jar, and it's a pretty essential plugin for
> >> just about every SBT project
> >> - Also there is no equivalent (AFAIK) to the maven shader plugin.
> >>
> >> I also wish that the dependency-graph plugin was included by default,
> >> but that's just me  :)
> >>
> >> -Evan
> >>
> >>
> >> On Fri, Mar 14, 2014 at 6:47 AM, jsuereth <joshua.suereth@gmail.com>
> wrote:
> >>> Hey guys -
> >>>
> >>> If there's anything we can do to improve the sbt experience, let me
> know.
> >>> I'd be extremely interested to see how/where there are issues
> integrating
> >>> sbt with the existing Hadoop ecosystem.
> >>>
> >>> Particularly the difficulties in using Sbt + Maven together (something
> which
> >>> tends block more than just spark from adopting sbt).
> >>>
> >>> I'm more than happy to listen and see what we can do on the sbt side
> to make
> >>> this as seamless as possible for all parties.
> >>>
> >>> Thanks!
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Necessity-of-Maven-and-SBT-Build-in-Spark-tp2315p5682.html
> >>> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
> >>
> >>
> >>
> >> --
> >> --
> >> Evan Chan
> >> Staff Engineer
> >> ev@ooyala.com  |
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message