spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivaram Venkataraman <shiva...@eecs.berkeley.edu>
Subject Re: Needs a matrix library
Date Sat, 07 Sep 2013 00:26:03 GMT
For the machine learning library that is a part of Spark 0.8 we have been
using jblas for local matrix operations. From some limited benchmarking
that we did, jblas is not much slower than optimized C++ libraries.
http://blog.mikiobraun.de/2009/04/some-benchmark-numbers-for-jblas.html has
some more details.

For more complex operations than addition and multiplication, mahout-math
is a pretty good library. There was a great discussion on pros/cons of
different Java/Scala-based matrix libraries in
https://github.com/mesos/spark/pull/736

Thanks
Shivaram


On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rxin@cs.berkeley.edu> wrote:

> They are asking about dedicated matrix libraries.
>
> Neither GraphX nor Giraph are matrix libraries. These are systems that
> handle large scale graph processing, which could possibly be modeled as
> matrix computations.  Hama looks like a BSP framework, so I am not sure if
> it has anything to do with matrix library either.
>
> For very small matrices (3x3, 4x4), the cost of going through jni to do
> native matrix operations will likely dominate the computation itself, so
> you are probably better off with a simple unrolled for loop in Java.
>
> I haven't looked into this myself, but I heard mahout-math is a decent
> library.
>
> --
> Reynold Xin, AMPLab, UC Berkeley
> http://rxin.org
>
>
>
> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> wrote:
>
> > keep forgetting this: what is graphx release roadmap?
> >
> > On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
> > > Would it be more logical to use GraphX ?
> > >   https://amplab.cs.berkeley.edu/publication/graphx-grades/
> > >
> > > Cos
> > >
> > > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> > >> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
> > >> wasn't sure it supported matrices). If Giraph supports matrices, big
> +1.
> > >>
> > >> Cheers,
> > >> Chris
> > >>
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> Chris Mattmann, Ph.D.
> > >> Senior Computer Scientist
> > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >> Office: 171-266B, Mailstop: 171-246
> > >> Email: chris.a.mattmann@nasa.gov
> > >> WWW:  http://sunset.usc.edu/~mattmann/
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> Adjunct Assistant Professor, Computer Science Department
> > >> University of Southern California, Los Angeles, CA 90089 USA
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Roman Shaposhnik <rvs@apache.org>
> > >> Date: Friday, September 6, 2013 2:00 PM
> > >> To: <dev@spark.incubator.apache.org>
> > >> Cc: "dev@sis.apache.org" <dev@sis.apache.org>
> > >> Subject: Re: Needs a matrix library
> > >>
> > >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> > >> ><chris.a.mattmann@jpl.nasa.gov> wrote:
> > >> >> Hey Martin,
> > >> >>
> > >> >> We may seriously consider using either Apache Hama here (which
will
> > >> >> bring in Hadoop):
> > >> >
> > >> >On that note I'd highly recommend taking a look at Apache Giraph
> > >> >as well: http://giraph.apache.org/
> > >> >
> > >> >Thanks,
> > >> >Roman.
> > >> >
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message