spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishna Sankar <ksanka...@gmail.com>
Subject Re: [VOTE] Release Apache Spark 1.2.0 (RC1)
Date Thu, 11 Dec 2014 05:55:14 GMT
   - K-Means iPython notebook & data attached.
   - It is the zip that gives the error ; while one of the RDDs is from the
   prediction, most probably there is no problem with the K-Means.
   - Lines 34,35 & 36 essentially are the same. But only 36 works with
   1.2.0.
   - Interestingly, lines 34,35 & 36 work with 1.1.1 (Checked just now)
      - The plot thickens!
      - In 1.1.1, freq_cluster_map.take(5) prints normally for 34 & 35, but
      in exponential form for 36. So there is some difference even in 1.1.1.
      - #34,#35 [(array([28143, 0, 174, 1, 0, 0, 7000]), 1),

       (array([19244,     0,   215,     2,     0,     0,  6968]), 1),
       (array([41354,     0,  4123,     4,     0,     0,  7034]), 1),
       (array([14776,     0,   500,     1,     0,     0,  6952]), 1),
       (array([97752,     0, 43300,    26,  2077,     4,  6935]), 0)]

      -

      #36 [(array([  2.81430000e+04,   0.00000000e+00,   1.74000000e+02,

                 1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
                 7.00000000e+03]), 1),
       (array([  1.92440000e+04,   0.00000000e+00,   2.15000000e+02,
                 2.00000000e+00,   0.00000000e+00,   0.00000000e+00,
                 6.96800000e+03]), 1),
       (array([  4.13540000e+04,   0.00000000e+00,   4.12300000e+03,
                 4.00000000e+00,   0.00000000e+00,   0.00000000e+00,
                 7.03400000e+03]), 1),
       (array([  1.47760000e+04,   0.00000000e+00,   5.00000000e+02,
                 1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
                 6.95200000e+03]), 1),
       (array([  9.77520000e+04,   0.00000000e+00,   4.33000000e+04,
                 2.60000000e+01,   2.07700000e+03,   4.00000000e+00,
                 6.93500000e+03]), 0)]

      - I had overwritten the naive bayes example. Will chase the older
   versions down

Cheers
<k/>

On Wed, Dec 3, 2014 at 4:19 PM, Xiangrui Meng <mengxr@gmail.com> wrote:

> Krishna, could you send me some code snippets for the issues you saw
> in naive Bayes and k-means? -Xiangrui
>
> On Sun, Nov 30, 2014 at 6:49 AM, Krishna Sankar <ksankar42@gmail.com>
> wrote:
> > +1
> > 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
> > -Dhadoop.version=2.4.0 -DskipTests clean package 16:46 min (slightly
> slower
> > connection)
> > 2. Tested pyspark, mlib - running as well as compare esults with 1.1.x
> > 2.1. statistics OK
> > 2.2. Linear/Ridge/Laso Regression OK
> >        Slight difference in the print method (vs. 1.1.x) of the model
> > object - with a label & more details. This is good.
> > 2.3. Decision Tree, Naive Bayes OK
> >        Changes in print(model) - now print (model.ToDebugString()) - OK
> >        Some changes in NaiveBayes. Different from my 1.1.x code - had to
> > flatten list structures, zip required same number in partitions
> >        After code changes ran fine.
> > 2.4. KMeans OK
> >        zip occasionally fails with error "localhost):
> > org.apache.spark.SparkException: Can only zip RDDs with same number of
> > elements in each partition"
> > Has https://issues.apache.org/jira/browse/SPARK-2251 reappeared ?
> > Made it work by doing a different transformation ie reusing an original
> > rdd.
> > 2.5. rdd operations OK
> >        State of the Union Texts - MapReduce, Filter,sortByKey (word
> count)
> > 2.6. recommendation OK
> > 2.7. Good work ! In 1.x.x, had a map distinct over the movielens medium
> > dataset which never worked. Works fine in 1.2.0 !
> > 3. Scala Mlib - subset of examples as in #2 above, with Scala
> > 3.1. statistics OK
> > 3.2. Linear Regression OK
> > 3.3. Decision Tree OK
> > 3.4. KMeans OK
> > Cheers
> > <k/>
> > P.S: Plan to add RF and .ml mechanics to this bank
> >
> > On Fri, Nov 28, 2014 at 9:16 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
> >
> >> Please vote on releasing the following candidate as Apache Spark version
> >> 1.2.0!
> >>
> >> The tag to be voted on is v1.2.0-rc1 (commit 1056e9ec1):
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=1056e9ec13203d0c51564265e94d77a054498fdb
> >>
> >> The release files, including signatures, digests, etc. can be found at:
> >> http://people.apache.org/~pwendell/spark-1.2.0-rc1/
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/pwendell.asc
> >>
> >> The staging repository for this release can be found at:
> >> https://repository.apache.org/content/repositories/orgapachespark-1048/
> >>
> >> The documentation corresponding to this release can be found at:
> >> http://people.apache.org/~pwendell/spark-1.2.0-rc1-docs/
> >>
> >> Please vote on releasing this package as Apache Spark 1.2.0!
> >>
> >> The vote is open until Tuesday, December 02, at 05:15 UTC and passes
> >> if a majority of at least 3 +1 PMC votes are cast.
> >>
> >> [ ] +1 Release this package as Apache Spark 1.1.0
> >> [ ] -1 Do not release this package because ...
> >>
> >> To learn more about Apache Spark, please see
> >> http://spark.apache.org/
> >>
> >> == What justifies a -1 vote for this release? ==
> >> This vote is happening very late into the QA period compared with
> >> previous votes, so -1 votes should only occur for significant
> >> regressions from 1.0.2. Bugs already present in 1.1.X, minor
> >> regressions, or bugs related to new features will not block this
> >> release.
> >>
> >> == What default changes should I be aware of? ==
> >> 1. The default value of "spark.shuffle.blockTransferService" has been
> >> changed to "netty"
> >> --> Old behavior can be restored by switching to "nio"
> >>
> >> 2. The default value of "spark.shuffle.manager" has been changed to
> "sort".
> >> --> Old behavior can be restored by setting "spark.shuffle.manager" to
> >> "hash".
> >>
> >> == Other notes ==
> >> Because this vote is occurring over a weekend, I will likely extend
> >> the vote if this RC survives until the end of the vote period.
> >>
> >> - Patrick
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: dev-help@spark.apache.org
> >>
> >>
>

Mime
View raw message