spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: Needs a matrix library
Date Sat, 07 Sep 2013 00:25:14 GMT
On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rxin@cs.berkeley.edu> wrote:
> They are asking about dedicated matrix libraries.

Ah. I did not read the quoted email. Not sure why Chris was talking
about Pregel stuff, that doesn't seem what that question was about.

>
> Neither GraphX nor Giraph are matrix libraries. These are systems that
> handle large scale graph processing, which could possibly be modeled as
> matrix computations.  Hama looks like a BSP framework, so I am not sure if
> it has anything to do with matrix library either.

+1
>
> For very small matrices (3x3, 4x4), the cost of going through jni to do
> native matrix operations will likely dominate the computation itself, so
> you are probably better off with a simple unrolled for loop in Java.

+1 i guess this note is about JBlas and JBlas-based derivatives like Breeze

>
> I haven't looked into this myself, but I heard mahout-math is a decent
> library.

+1 although for such tiny things like 3x3, 4x4  our cost-based
optimizations are probably not going to provide any noticeable bang.
Mahout in-core math is mostly for uniform cost-optimized support of
sparse vectors along with dense.

Also, see if this makes sense, we are leaning towards commiting these
scala mappings in the current Mahout's trunk :[1]

[1] http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in-core-linear.html

-Dmitriy

>
> --
> Reynold Xin, AMPLab, UC Berkeley
> http://rxin.org
>
>
>
> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:
>
>> keep forgetting this: what is graphx release roadmap?
>>
>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <cos@apache.org> wrote:
>> > Would it be more logical to use GraphX ?
>> >   https://amplab.cs.berkeley.edu/publication/graphx-grades/
>> >
>> > Cos
>> >
>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs but
>> >> wasn't sure it supported matrices). If Giraph supports matrices, big +1.
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Chris Mattmann, Ph.D.
>> >> Senior Computer Scientist
>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> Office: 171-266B, Mailstop: 171-246
>> >> Email: chris.a.mattmann@nasa.gov
>> >> WWW:  http://sunset.usc.edu/~mattmann/
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Adjunct Assistant Professor, Computer Science Department
>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Roman Shaposhnik <rvs@apache.org>
>> >> Date: Friday, September 6, 2013 2:00 PM
>> >> To: <dev@spark.incubator.apache.org>
>> >> Cc: "dev@sis.apache.org" <dev@sis.apache.org>
>> >> Subject: Re: Needs a matrix library
>> >>
>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>> >> ><chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> >> Hey Martin,
>> >> >>
>> >> >> We may seriously consider using either Apache Hama here (which
will
>> >> >> bring in Hadoop):
>> >> >
>> >> >On that note I'd highly recommend taking a look at Apache Giraph
>> >> >as well: http://giraph.apache.org/
>> >> >
>> >> >Thanks,
>> >> >Roman.
>> >> >
>> >>
>>

Mime
View raw message