spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc2)
Date Mon, 09 Dec 2013 04:42:18 GMT
And I think 195 is sufficient to build something that works; but I haven't
personally tested it.


On Sun, Dec 8, 2013 at 8:41 PM, Mark Hamstra <mark@clearstorydata.com>wrote:

> Well, what I've already done for ClearStory is very close to how Debian
> packaging should be done for Apache Spark.  That much can be put into a
> pull request quickly.  The only real issues are exactly how the packages
> should be named, checking that the metadata of the packages are exactly
> correct for an Apache release, and deciding whether we should be producing
> a spark-examples package, spark-tools package, separate the Java and Python
> APIs into their own packages, create a source package, etc.  What we've
> been doing up to now is essentially just the minimal packaging of a fat jar
> that can be deployed by Chef or something similar.  It's never really been
> put together in a way appropriate to go into a Debian or Ubuntu
> distribution, for instance.
>
>
> On Sun, Dec 8, 2013 at 8:25 PM, Patrick Wendell <pwendell@gmail.com>wrote:
>
>> Hey Mark,
>>
>> Okay if 195 gets this in working order in the branch 0.8 let's just
>> merge that to keep it consistent with our docs and the way this is
>> done in 0.8.0
>>
>> We can do a broader refactoring in 0.9. Would be great if you could
>> kick off a JIRA discussion or submit a PR relating to that.
>>
>> - Patrick
>>
>> On Sun, Dec 8, 2013 at 8:07 PM, Mark Hamstra <mark@clearstorydata.com>
>> wrote:
>> > Well, 195 is sufficient to give you something that runs, but it doesn't
>> run
>> > the same way as Spark built/distributed by other means -- e.g., after
>> 195
>> > the package still uses something equivalent to the old `run` script
>> instead
>> > of the current `spark-class` way.
>> >
>> >
>> > On Sun, Dec 8, 2013 at 8:02 PM, Patrick Wendell <pwendell@gmail.com>
>> wrote:
>> >
>> >> Hey Mark,
>> >>
>> >> What I'm asking is whether this patch is sufficient to have a working
>> >> debian build in 0.8.1, or are there other outstanding issues to make
>> >> it work? By working I mean, within the initial design that was
>> >> contributed (with repl-bin) it works according to that approach.
>> >>
>> >> We can redesign this packaging in 0.9. That will require having a PR
>> >> against Apache Spark, discussing, etc. But it doesn't need to be on
>> >> the critical path for this release.
>> >>
>> >> - Patrick
>> >>
>> >> On Sun, Dec 8, 2013 at 7:54 PM, Mark Hamstra <mark@clearstorydata.com>
>> >> wrote:
>> >> > Whatever Debian package gets built has to work, so that's the first
>> >> > requirement.  I don't know how to decide whether a change is
>> acceptable
>> >> in
>> >> > 0.8 or has to wait until 0.9, but the 0.9 packaging should definitely
>> >> > leverage the assembly sub-project, making repl-bin unnecessary.
>> >> >
>> >> >
>> >> > On Sun, Dec 8, 2013 at 7:46 PM, Patrick Wendell <pwendell@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Looked into this a bit more - I think removing repl-bin is something
>> >> >> we should wait until 0.9 to do, because we've published it to maven
>> in
>> >> >> 0.8.0 and people might expect it to be there in 0.8.1.
>> >> >>
>> >> >> Merging the directly referenced pull request (195) seems like a
good
>> >> >> idea though since it fixes a bug in the script.
>> >> >>
>> >> >> Is that what you are suggesting?
>> >> >>
>> >> >> - Patrick
>> >> >>
>> >> >> On Sun, Dec 8, 2013 at 7:30 PM, Patrick Wendell <pwendell@gmail.com
>> >
>> >> >> wrote:
>> >> >> > Hey Mark - ya this would be good to get in.
>> >> >> >
>> >> >> > Does merging that particular PR put this in sufficient shape
for
>> the
>> >> >> > 0.8.1 release or are there other open patches we need to look
at?
>> >> >> >
>> >> >> > - Patrick
>> >> >> >
>> >> >> > On Sun, Dec 8, 2013 at 6:05 PM, Mark Hamstra <
>> mark@clearstorydata.com
>> >> >
>> >> >> wrote:
>> >> >> >> SPARK-962 should be resolved before release.  See also:
>> >> >> >> https://github.com/apache/incubator-spark/pull/195
>> >> >> >>
>> >> >> >> With the references to the way I changed Debian packaging
for
>> >> >> ClearStory,
>> >> >> >> we should be at least 90% of the way toward doing it right
for
>> >> Apache.
>> >> >> >>
>> >> >> >>
>> >> >> >> On Sun, Dec 8, 2013 at 5:29 PM, Patrick Wendell <
>> pwendell@gmail.com>
>> >> >> wrote:
>> >> >> >>
>> >> >> >>> For my own part I'll give a +1 to this RC.
>> >> >> >>>
>> >> >> >>> On Sun, Dec 8, 2013 at 4:30 PM, Taka Shinagawa <
>> >> taka.epsilon@gmail.com
>> >> >> >
>> >> >> >>> wrote:
>> >> >> >>> > OK. I will post the entire output via separate
email. I just
>> >> upgraded
>> >> >> >>> > Hadoop to 2.2.0 recently. So there might be something
I need
>> to
>> >> >> >>> > remove/clean up.
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > On Sun, Dec 8, 2013 at 4:24 PM, Patrick Wendell
<
>> >> pwendell@gmail.com>
>> >> >> >>> wrote:
>> >> >> >>> >
>> >> >> >>> >> Hey Take,
>> >> >> >>> >>
>> >> >> >>> >> Could you start a separate thread to debug
your build issue?
>> In
>> >> that
>> >> >> >>> >> thread, could you paste the exact build command
and entire
>> >> output?
>> >> >> The
>> >> >> >>> log
>> >> >> >>> >> you posted here suggests the first build
detected hadoop
>> 1.0.4
>> >> not
>> >> >> 2.2.0
>> >> >> >>> >> based on the assembly file name it is logging.
>> >> >> >>> >>
>> >> >> >>> >> ---
>> >> >> >>> >> sent from my phone
>> >> >> >>> >> On Dec 8, 2013 4:13 PM, "Taka Shinagawa"
<
>> taka.epsilon@gmail.com
>> >> >
>> >> >> >>> wrote:
>> >> >> >>> >>
>> >> >> >>> >> > With Hadoop 2.2.0 (& Java 1.7.0_45)
installed, I'm having
>> >> trouble
>> >> >> >>> >> > completing the build process (sbt/sbt
assembly) on
>> Macbook. The
>> >> >> sbt
>> >> >> >>> >> command
>> >> >> >>> >> > hangs at the last step.
>> >> >> >>> >> >
>> >> >> >>> >> > ...
>> >> >> >>> >> > ...
>> >> >> >>> >> > [info] SHA-1: ce8275f5841002164c4305c912a2892ec7c1d395
>> >> >> >>> >> > [info] Packaging
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >>
>> >> >> >>>
>> >> >>
>> >>
>> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/tools/target/scala-2.9.3/spark-tools-assembly-0.8.1-incubating.jar
>> >> >> >>> >> > ...
>> >> >> >>> >> > [info] SHA-1: 0657a347240266230247693f265a5797d40c326a
>> >> >> >>> >> > [info] Packaging
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >>
>> >> >> >>>
>> >> >>
>> >>
>> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop1.0.4.jar
>> >> >> >>> >> > ...
>> >> >> >>> >> > (hangs here)
>> >> >> >>> >> > --------------------------
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > On another Macbook with Hadoop 1.1.1
(& Java 1.7.0_45)
>> >> installed,
>> >> >> I
>> >> >> >>> was
>> >> >> >>> >> > able to build it successfully.
>> >> >> >>> >> > ..
>> >> >> >>> >> > ..
>> >> >> >>> >> > [info] SHA-1: 77109cd085bd4f0d2b601b3451b35b961d357534
>> >> >> >>> >> > [info] Packaging
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >>
>> >> >> >>>
>> >> >>
>> >>
>> /Users/tshinagawa/Documents/Spark/RCs/spark-0.8.1-incubating/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar
>> >> >> >>> >> > ...
>> >> >> >>> >> > [info] Done packaging.
>> >> >> >>> >> > [success] Total time: 266 s, completed
Dec 8, 2013 3:03:10
>> PM
>> >> >> >>> >> > --------------------------
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > On Sun, Dec 8, 2013 at 12:41 PM, Patrick
Wendell <
>> >> >> pwendell@gmail.com>
>> >> >> >>> >> > wrote:
>> >> >> >>> >> >
>> >> >> >>> >> > > Please vote on releasing the following
candidate as
>> Apache
>> >> Spark
>> >> >> >>> >> > > (incubating) version 0.8.1.
>> >> >> >>> >> > >
>> >> >> >>> >> > > The tag to be voted on is v0.8.1-incubating
(commit
>> >> bf23794a):
>> >> >> >>> >> > >
>> >> >> >>> >> > >
>> >> >> >>> >> >
>> >> >> >>> >>
>> >> >> >>>
>> >> >>
>> >>
>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=tag;h=e6ba91b5a7527316202797fc3dce469ff86cf203
>> >> >> >>> >> > >
>> >> >> >>> >> > > The release files, including signatures,
digests, etc
>> can be
>> >> >> found
>> >> >> >>> at:
>> >> >> >>> >> > >
>> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2/
>> >> >> >>> >> > >
>> >> >> >>> >> > > Release artifacts are signed with
the following key:
>> >> >> >>> >> > > https://people.apache.org/keys/committer/pwendell.asc
>> >> >> >>> >> > >
>> >> >> >>> >> > > The staging repository for this
release can be found at:
>> >> >> >>> >> > >
>> >> >> >>>
>> >> https://repository.apache.org/content/repositories/orgapachespark-024/
>> >> >> >>> >> > >
>> >> >> >>> >> > > The documentation corresponding
to this release can be
>> found
>> >> at:
>> >> >> >>> >> > >
>> >> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2-docs/
>> >> >> >>> >> > >
>> >> >> >>> >> > > For information about the contents
of this release see:
>> >> >> >>> >> > > <attached> draft of release
notes
>> >> >> >>> >> > > <attached> draft of release
credits
>> >> >> >>> >> > >
>> >> >> >>>
>> >> https://github.com/apache/incubator-spark/blob/branch-0.8/CHANGES.txt
>> >> >> >>> >> > >
>> >> >> >>> >> > > Please vote on releasing this package
as Apache Spark
>> >> >> >>> 0.8.1-incubating!
>> >> >> >>> >> > >
>> >> >> >>> >> > > The vote is open until Wednesday,
December 11th at 21:00
>> UTC
>> >> and
>> >> >> >>> >> > > passes if a majority of at least
3 +1 PPMC votes are
>> cast.
>> >> >> >>> >> > >
>> >> >> >>> >> > > [ ] +1 Release this package as
Apache Spark
>> 0.8.1-incubating
>> >> >> >>> >> > > [ ] -1 Do not release this package
because ...
>> >> >> >>> >> > >
>> >> >> >>> >> > > To learn more about Apache Spark,
please see
>> >> >> >>> >> > > http://spark.incubator.apache.org/
>> >> >> >>> >> > >
>> >> >> >>> >> >
>> >> >> >>> >>
>> >> >> >>>
>> >> >>
>> >>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message