spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc2)
Date Mon, 09 Dec 2013 04:41:23 GMT
Well, what I've already done for ClearStory is very close to how Debian
packaging should be done for Apache Spark.  That much can be put into a
pull request quickly.  The only real issues are exactly how the packages
should be named, checking that the metadata of the packages are exactly
correct for an Apache release, and deciding whether we should be producing
a spark-examples package, spark-tools package, separate the Java and Python
APIs into their own packages, create a source package, etc.  What we've
been doing up to now is essentially just the minimal packaging of a fat jar
that can be deployed by Chef or something similar.  It's never really been
put together in a way appropriate to go into a Debian or Ubuntu
distribution, for instance.


On Sun, Dec 8, 2013 at 8:25 PM, Patrick Wendell <pwendell@gmail.com> wrote:

> Hey Mark,
>
> Okay if 195 gets this in working order in the branch 0.8 let's just
> merge that to keep it consistent with our docs and the way this is
> done in 0.8.0
>
> We can do a broader refactoring in 0.9. Would be great if you could
> kick off a JIRA discussion or submit a PR relating to that.
>
> - Patrick
>
> On Sun, Dec 8, 2013 at 8:07 PM, Mark Hamstra <mark@clearstorydata.com>
> wrote:
> > Well, 195 is sufficient to give you something that runs, but it doesn't
> run
> > the same way as Spark built/distributed by other means -- e.g., after 195
> > the package still uses something equivalent to the old `run` script
> instead
> > of the current `spark-class` way.
> >
> >
> > On Sun, Dec 8, 2013 at 8:02 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
> >
> >> Hey Mark,
> >>
> >> What I'm asking is whether this patch is sufficient to have a working
> >> debian build in 0.8.1, or are there other outstanding issues to make
> >> it work? By working I mean, within the initial design that was
> >> contributed (with repl-bin) it works according to that approach.
> >>
> >> We can redesign this packaging in 0.9. That will require having a PR
> >> against Apache Spark, discussing, etc. But it doesn't need to be on
> >> the critical path for this release.
> >>
> >> - Patrick
> >>
> >> On Sun, Dec 8, 2013 at 7:54 PM, Mark Hamstra <mark@clearstorydata.com>
> >> wrote:
> >> > Whatever Debian package gets built has to work, so that's the first
> >> > requirement.  I don't know how to decide whether a change is
> acceptable
> >> in
> >> > 0.8 or has to wait until 0.9, but the 0.9 packaging should definitely
> >> > leverage the assembly sub-project, making repl-bin unnecessary.
> >> >
> >> >
> >> > On Sun, Dec 8, 2013 at 7:46 PM, Patrick Wendell <pwendell@gmail.com>
> >> wrote:
> >> >
> >> >> Looked into this a bit more - I think removing repl-bin is something
> >> >> we should wait until 0.9 to do, because we've published it to maven
> in
> >> >> 0.8.0 and people might expect it to be there in 0.8.1.
> >> >>
> >> >> Merging the directly referenced pull request (195) seems like a good
> >> >> idea though since it fixes a bug in the script.
> >> >>
> >> >> Is that what you are suggesting?
> >> >>
> >> >> - Patrick
> >> >>
> >> >> On Sun, Dec 8, 2013 at 7:30 PM, Patrick Wendell <pwendell@gmail.com>
> >> >> wrote:
> >> >> > Hey Mark - ya this would be good to get in.
> >> >> >
> >> >> > Does merging that particular PR put this in sufficient shape for
> the
> >> >> > 0.8.1 release or are there other open patches we need to look
at?
> >> >> >
> >> >> > - Patrick
> >> >> >
> >> >> > On Sun, Dec 8, 2013 at 6:05 PM, Mark Hamstra <
> mark@clearstorydata.com
> >> >
> >> >> wrote:
> >> >> >> SPARK-962 should be resolved before release.  See also:
> >> >> >> https://github.com/apache/incubator-spark/pull/195
> >> >> >>
> >> >> >> With the references to the way I changed Debian packaging
for
> >> >> ClearStory,
> >> >> >> we should be at least 90% of the way toward doing it right
for
> >> Apache.
> >> >> >>
> >> >> >>
> >> >> >> On Sun, Dec 8, 2013 at 5:29 PM, Patrick Wendell <
> pwendell@gmail.com>
> >> >> wrote:
> >> >> >>
> >> >> >>> For my own part I'll give a +1 to this RC.
> >> >> >>>
> >> >> >>> On Sun, Dec 8, 2013 at 4:30 PM, Taka Shinagawa <
> >> taka.epsilon@gmail.com
> >> >> >
> >> >> >>> wrote:
> >> >> >>> > OK. I will post the entire output via separate email.
I just
> >> upgraded
> >> >> >>> > Hadoop to 2.2.0 recently. So there might be something
I need to
> >> >> >>> > remove/clean up.
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > On Sun, Dec 8, 2013 at 4:24 PM, Patrick Wendell <
> >> pwendell@gmail.com>
> >> >> >>> wrote:
> >> >> >>> >
> >> >> >>> >> Hey Take,
> >> >> >>> >>
> >> >> >>> >> Could you start a separate thread to debug your
build issue?
> In
> >> that
> >> >> >>> >> thread, could you paste the exact build command
and entire
> >> output?
> >> >> The
> >> >> >>> log
> >> >> >>> >> you posted here suggests the first build detected
hadoop 1.0.4
> >> not
> >> >> 2.2.0
> >> >> >>> >> based on the assembly file name it is logging.
> >> >> >>> >>
> >> >> >>> >> ---
> >> >> >>> >> sent from my phone
> >> >> >>> >> On Dec 8, 2013 4:13 PM, "Taka Shinagawa" <
> taka.epsilon@gmail.com
> >> >
> >> >> >>> wrote:
> >> >> >>> >>
> >> >> >>> >> > With Hadoop 2.2.0 (& Java 1.7.0_45)
installed, I'm having
> >> trouble
> >> >> >>> >> > completing the build process (sbt/sbt assembly)
on Macbook.
> The
> >> >> sbt
> >> >> >>> >> command
> >> >> >>> >> > hangs at the last step.
> >> >> >>> >> >
> >> >> >>> >> > ...
> >> >> >>> >> > ...
> >> >> >>> >> > [info] SHA-1: ce8275f5841002164c4305c912a2892ec7c1d395
> >> >> >>> >> > [info] Packaging
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >>
> >> >> >>>
> >> >>
> >>
> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/tools/target/scala-2.9.3/spark-tools-assembly-0.8.1-incubating.jar
> >> >> >>> >> > ...
> >> >> >>> >> > [info] SHA-1: 0657a347240266230247693f265a5797d40c326a
> >> >> >>> >> > [info] Packaging
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >>
> >> >> >>>
> >> >>
> >>
> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop1.0.4.jar
> >> >> >>> >> > ...
> >> >> >>> >> > (hangs here)
> >> >> >>> >> > --------------------------
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > On another Macbook with Hadoop 1.1.1 (&
Java 1.7.0_45)
> >> installed,
> >> >> I
> >> >> >>> was
> >> >> >>> >> > able to build it successfully.
> >> >> >>> >> > ..
> >> >> >>> >> > ..
> >> >> >>> >> > [info] SHA-1: 77109cd085bd4f0d2b601b3451b35b961d357534
> >> >> >>> >> > [info] Packaging
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >>
> >> >> >>>
> >> >>
> >>
> /Users/tshinagawa/Documents/Spark/RCs/spark-0.8.1-incubating/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar
> >> >> >>> >> > ...
> >> >> >>> >> > [info] Done packaging.
> >> >> >>> >> > [success] Total time: 266 s, completed Dec
8, 2013 3:03:10
> PM
> >> >> >>> >> > --------------------------
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > On Sun, Dec 8, 2013 at 12:41 PM, Patrick
Wendell <
> >> >> pwendell@gmail.com>
> >> >> >>> >> > wrote:
> >> >> >>> >> >
> >> >> >>> >> > > Please vote on releasing the following
candidate as Apache
> >> Spark
> >> >> >>> >> > > (incubating) version 0.8.1.
> >> >> >>> >> > >
> >> >> >>> >> > > The tag to be voted on is v0.8.1-incubating
(commit
> >> bf23794a):
> >> >> >>> >> > >
> >> >> >>> >> > >
> >> >> >>> >> >
> >> >> >>> >>
> >> >> >>>
> >> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=tag;h=e6ba91b5a7527316202797fc3dce469ff86cf203
> >> >> >>> >> > >
> >> >> >>> >> > > The release files, including signatures,
digests, etc can
> be
> >> >> found
> >> >> >>> at:
> >> >> >>> >> > >
> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2/
> >> >> >>> >> > >
> >> >> >>> >> > > Release artifacts are signed with the
following key:
> >> >> >>> >> > > https://people.apache.org/keys/committer/pwendell.asc
> >> >> >>> >> > >
> >> >> >>> >> > > The staging repository for this release
can be found at:
> >> >> >>> >> > >
> >> >> >>>
> >> https://repository.apache.org/content/repositories/orgapachespark-024/
> >> >> >>> >> > >
> >> >> >>> >> > > The documentation corresponding to
this release can be
> found
> >> at:
> >> >> >>> >> > >
> >> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2-docs/
> >> >> >>> >> > >
> >> >> >>> >> > > For information about the contents
of this release see:
> >> >> >>> >> > > <attached> draft of release notes
> >> >> >>> >> > > <attached> draft of release credits
> >> >> >>> >> > >
> >> >> >>>
> >> https://github.com/apache/incubator-spark/blob/branch-0.8/CHANGES.txt
> >> >> >>> >> > >
> >> >> >>> >> > > Please vote on releasing this package
as Apache Spark
> >> >> >>> 0.8.1-incubating!
> >> >> >>> >> > >
> >> >> >>> >> > > The vote is open until Wednesday, December
11th at 21:00
> UTC
> >> and
> >> >> >>> >> > > passes if a majority of at least 3
+1 PPMC votes are cast.
> >> >> >>> >> > >
> >> >> >>> >> > > [ ] +1 Release this package as Apache
Spark
> 0.8.1-incubating
> >> >> >>> >> > > [ ] -1 Do not release this package
because ...
> >> >> >>> >> > >
> >> >> >>> >> > > To learn more about Apache Spark, please
see
> >> >> >>> >> > > http://spark.incubator.apache.org/
> >> >> >>> >> > >
> >> >> >>> >> >
> >> >> >>> >>
> >> >> >>>
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message