spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (398J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Needs a matrix library
Date Sat, 07 Sep 2013 01:41:46 GMT
Thanks guys, just sharing a need here. SIS is a fully Java based
geospatial library in development at Apache, aiming to support OGC
standards. It would be great to figure out some synergy between Spark/Shark
and SIS..

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Dmitriy Lyubimov <dlieu.7@gmail.com>
Reply-To: "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org>
Date: Friday, September 6, 2013 5:25 PM
To: "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org>
Cc: "dev@sis.apache.org" <dev@sis.apache.org>
Subject: Re: Needs a matrix library

>On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rxin@cs.berkeley.edu> wrote:
>> They are asking about dedicated matrix libraries.
>
>Ah. I did not read the quoted email. Not sure why Chris was talking
>about Pregel stuff, that doesn't seem what that question was about.
>
>>
>> Neither GraphX nor Giraph are matrix libraries. These are systems that
>> handle large scale graph processing, which could possibly be modeled as
>> matrix computations.  Hama looks like a BSP framework, so I am not sure
>>if
>> it has anything to do with matrix library either.
>
>+1
>>
>> For very small matrices (3x3, 4x4), the cost of going through jni to do
>> native matrix operations will likely dominate the computation itself, so
>> you are probably better off with a simple unrolled for loop in Java.
>
>+1 i guess this note is about JBlas and JBlas-based derivatives like
>Breeze
>
>>
>> I haven't looked into this myself, but I heard mahout-math is a decent
>> library.
>
>+1 although for such tiny things like 3x3, 4x4  our cost-based
>optimizations are probably not going to provide any noticeable bang.
>Mahout in-core math is mostly for uniform cost-optimized support of
>sparse vectors along with dense.
>
>Also, see if this makes sense, we are leaning towards commiting these
>scala mappings in the current Mahout's trunk :[1]
>
>[1] 
>http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in
>-core-linear.html
>
>-Dmitriy
>
>>
>> --
>> Reynold Xin, AMPLab, UC Berkeley
>> http://rxin.org
>>
>>
>>
>> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dlieu.7@gmail.com>
>>wrote:
>>
>>> keep forgetting this: what is graphx release roadmap?
>>>
>>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <cos@apache.org>
>>>wrote:
>>> > Would it be more logical to use GraphX ?
>>> >   https://amplab.cs.berkeley.edu/publication/graphx-grades/
>>> >
>>> > Cos
>>> >
>>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
>>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
>>>but
>>> >> wasn't sure it supported matrices). If Giraph supports matrices,
>>>big +1.
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW:  http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Roman Shaposhnik <rvs@apache.org>
>>> >> Date: Friday, September 6, 2013 2:00 PM
>>> >> To: <dev@spark.incubator.apache.org>
>>> >> Cc: "dev@sis.apache.org" <dev@sis.apache.org>
>>> >> Subject: Re: Needs a matrix library
>>> >>
>>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
>>> >> ><chris.a.mattmann@jpl.nasa.gov> wrote:
>>> >> >> Hey Martin,
>>> >> >>
>>> >> >> We may seriously consider using either Apache Hama here (which
>>>will
>>> >> >> bring in Hadoop):
>>> >> >
>>> >> >On that note I'd highly recommend taking a look at Apache Giraph
>>> >> >as well: http://giraph.apache.org/
>>> >> >
>>> >> >Thanks,
>>> >> >Roman.
>>> >> >
>>> >>
>>>


Mime
View raw message