spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)
Date Wed, 11 Dec 2013 21:10:34 GMT
I don't know how to make sense of the numbers, but here's what I've got
from a very small sample size.

For both v0.8.0-incubating and v0.8.1-incubating, building separate
assemblies is faster than `./sbt/sbt assembly` and the times for building
separate assemblies for 0.8.0 and 0.8.1 are about the same.

For v0.8.0-incubating, `./sbt/sbt assembly` takes about 2.5x as long as the
sum of the separate assemblies.
For v0.8.1-incubating, `./sbt/sbt assembly` takes almost 8x as long as the
sum of the separate assemblies.

Weird.


On Wed, Dec 11, 2013 at 11:49 AM, Patrick Wendell <pwendell@gmail.com>wrote:

> I'll +1 myself also.
>
> For anyone who has the slow build problem: does this issue happen when
> building v0.8.0-incubating also? Trying to figure out whether it's
> related to something we added in 0.8.1 or if it's a long standing
> issue.
>
> - Patrick
>
> On Wed, Dec 11, 2013 at 10:39 AM, Matei Zaharia <matei.zaharia@gmail.com>
> wrote:
> > Woah, weird, but definitely good to know.
> >
> > If you’re doing Spark development, there’s also a more convenient option
> added by Shivaram in the master branch. You can do sbt assemble-deps to
> package *just* the dependencies of each project in a special assembly JAR,
> and then use sbt compile to update the code. This will use the classes
> directly out of the target/scala-2.9.3/classes directories. You have to
> redo assemble-deps only if your external dependencies change.
> >
> > Matei
> >
> > On Dec 11, 2013, at 1:04 AM, Prashant Sharma <scrapcodes@gmail.com>
> wrote:
> >
> >> I hope this PR https://github.com/apache/incubator-spark/pull/252 can
> help.
> >> Again this is not a blocker for the release from my side either.
> >>
> >>
> >> On Wed, Dec 11, 2013 at 2:14 PM, Mark Hamstra <mark@clearstorydata.com
> >wrote:
> >>
> >>> Interesting, and confirmed: On my machine where `./sbt/sbt assembly`
> takes
> >>> a long, long, looooong time to complete (a MBP, in my case), building
> three
> >>> separate assemblies (`./sbt/sbt assembly/assembly`, `./sbt/sbt
> >>> examples/assembly`, `./sbt/sbt tools/assembly`) takes much, much less
> time.
> >>>
> >>>
> >>>
> >>> On Wed, Dec 11, 2013 at 12:02 AM, Prashant Sharma <
> scrapcodes@gmail.com
> >>>> wrote:
> >>>
> >>>> forgot to mention, after running sbt/sbt assembly/assembly running
> >>> sbt/sbt
> >>>> examples/assembly takes just 37s. Not to mention my hardware is not
> >>> really
> >>>> great.
> >>>>
> >>>>
> >>>> On Wed, Dec 11, 2013 at 1:28 PM, Prashant Sharma <
> scrapcodes@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> Hi Patrick and Matei,
> >>>>>
> >>>>> Was trying out this and followed the quick start guide which says
do
> >>>>> sbt/sbt assembly, like few others I was also stuck for few minutes
on
> >>>>> linux. On the other hand if I use sbt/sbt assembly/assembly it is
> much
> >>>>> faster.
> >>>>>
> >>>>> Should we change the documentation to reflect this. It will not
be
> >>> great
> >>>>> for first time users to get stuck there.
> >>>>>
> >>>>>
> >>>>> On Wed, Dec 11, 2013 at 9:54 AM, Matei Zaharia <
> >>> matei.zaharia@gmail.com
> >>>>> wrote:
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> Built and tested it on Mac OS X.
> >>>>>>
> >>>>>> Matei
> >>>>>>
> >>>>>>
> >>>>>> On Dec 10, 2013, at 4:49 PM, Patrick Wendell <pwendell@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> Please vote on releasing the following candidate as Apache
Spark
> >>>>>>> (incubating) version 0.8.1.
> >>>>>>>
> >>>>>>> The tag to be voted on is v0.8.1-incubating (commit b87d31d):
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=commit;h=b87d31dd8eb4b4e47c0138e9242d0dd6922c8c4e
> >>>>>>>
> >>>>>>> The release files, including signatures, digests, etc can
be found
> >>> at:
> >>>>>>> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4/
> >>>>>>>
> >>>>>>> Release artifacts are signed with the following key:
> >>>>>>> https://people.apache.org/keys/committer/pwendell.asc
> >>>>>>>
> >>>>>>> The staging repository for this release can be found at:
> >>>>>>>
> >>>>
> https://repository.apache.org/content/repositories/orgapachespark-040/
> >>>>>>>
> >>>>>>> The documentation corresponding to this release can be found
at:
> >>>>>>>
> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4-docs/
> >>>>>>>
> >>>>>>> For information about the contents of this release see:
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=blob;f=CHANGES.txt;h=ce0aeab524505b63c7999e0371157ac2def6fe1c;hb=branch-0.8
> >>>>>>>
> >>>>>>> Please vote on releasing this package as Apache Spark
> >>>> 0.8.1-incubating!
> >>>>>>>
> >>>>>>> The vote is open until Saturday, December 14th at 01:00
UTC and
> >>>>>>> passes if a majority of at least 3 +1 PPMC votes are cast.
> >>>>>>>
> >>>>>>> [ ] +1 Release this package as Apache Spark 0.8.1-incubating
> >>>>>>> [ ] -1 Do not release this package because ...
> >>>>>>>
> >>>>>>> To learn more about Apache Spark, please see
> >>>>>>> http://spark.incubator.apache.org/
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> s
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> s
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> s
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message