spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Build Changes for SBT Users
Date Sat, 04 Jan 2014 21:20:55 GMT
That makes sense, I think we could structure a script in such a way that it
would overcome these problems though and probably provide a fair a mount of
benefit for people who just want to get started quickly.

The easiest would be to have it use the system sbt if present and then fall
back to downloading the sbt jar. As far as stability of the URL goes we
could solve this by either having it point at a domain we control, or just
with an clear error message indicating it failed to download sbt and the
user needs to install sbt.

If a restructured script in that manner would be useful I could whip up a
pull request :)


On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell <pwendell@gmail.com> wrote:

> We thought about this but elected not to do this for a few reasons.
>
> 1. Some people build from machines that do not have internet access
> for security reasons and retrieve dependency from internal nexus
> repositories. So having a build dependency that relies on internet
> downloads is not desirable.
>
> 2. It's a hard to ensure stability of a particular URL in perpetuity.
> This is why maven central and other mirror networks exist. Keep in
> mind that we can't change the release code ever once we release it,
> and if something changed about the particular URL it could break the
> build.
>
> - Patrick
>
> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <andrew@andrewash.com> wrote:
> > +1 on bundling a script similar to that one
> >
> >
> > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <holden@pigscanfly.ca>
> wrote:
> >
> >> Could we ship a shell script which downloads the sbt jar if not present
> >> (like for example https://github.com/holdenk/slashem/blob/master/sbt )?
> >>
> >>
> >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <pwendell@gmail.com>
> >> wrote:
> >>
> >> > Hey All,
> >> >
> >> > Due to an ASF requirement, we recently merged a patch which removes
> >> > the sbt jar from the build. This is necessary because we aren't
> >> > allowed to distributed binary artifacts with our source packages.
> >> >
> >> > This means that instead of building Spark with "sbt/sbt XXX", you'll
> >> > need to have sbt yourself and just run "sbt XXX" from within the Spark
> >> > directory. This is similar to the maven build, where we expect users
> >> > already have maven installed.
> >> >
> >> > You can download sbt at http://www.scala-sbt.org/. It's okay to just
> >> > download the most recent version of sbt, since sbt knows how to fetch
> >> > other versions of itself and will always use the one we specify in our
> >> > build file to compile spark.
> >> >
> >> > - Patrick
> >> >
> >>
> >>
> >>
> >> --
> >> Cell : 425-233-8271
> >>
>



-- 
Cell : 425-233-8271

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message