spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Estrada <estrada.a...@gmail.com>
Subject Re: Needs a matrix library
Date Sat, 07 Sep 2013 03:27:14 GMT
I agree with that sentiment, Dr. Mattmann! It would be extremely cool to
see the distributed computation communities (eg. Spark and Hadoop) take
advantage of SIS. This is especially true for processing geospatial vector
data. Geospatial raster data is very splittable which makes it ideal for
this type of batch processing. Vector data is another beast all together
and I encourage folks in the aforementioned communities to think about how
to do this. I certainly have ideas and am all ears if someone would like to
chat about it!

Regards,
Adam


On Fri, Sep 6, 2013 at 9:41 PM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Thanks guys, just sharing a need here. SIS is a fully Java based
> geospatial library in development at Apache, aiming to support OGC
> standards. It would be great to figure out some synergy between Spark/Shark
> and SIS..
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Dmitriy Lyubimov <dlieu.7@gmail.com>
> Reply-To: "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org
> >
> Date: Friday, September 6, 2013 5:25 PM
> To: "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org>
> Cc: "dev@sis.apache.org" <dev@sis.apache.org>
> Subject: Re: Needs a matrix library
>
> >On Fri, Sep 6, 2013 at 5:09 PM, Reynold Xin <rxin@cs.berkeley.edu> wrote:
> >> They are asking about dedicated matrix libraries.
> >
> >Ah. I did not read the quoted email. Not sure why Chris was talking
> >about Pregel stuff, that doesn't seem what that question was about.
> >
> >>
> >> Neither GraphX nor Giraph are matrix libraries. These are systems that
> >> handle large scale graph processing, which could possibly be modeled as
> >> matrix computations.  Hama looks like a BSP framework, so I am not sure
> >>if
> >> it has anything to do with matrix library either.
> >
> >+1
> >>
> >> For very small matrices (3x3, 4x4), the cost of going through jni to do
> >> native matrix operations will likely dominate the computation itself, so
> >> you are probably better off with a simple unrolled for loop in Java.
> >
> >+1 i guess this note is about JBlas and JBlas-based derivatives like
> >Breeze
> >
> >>
> >> I haven't looked into this myself, but I heard mahout-math is a decent
> >> library.
> >
> >+1 although for such tiny things like 3x3, 4x4  our cost-based
> >optimizations are probably not going to provide any noticeable bang.
> >Mahout in-core math is mostly for uniform cost-optimized support of
> >sparse vectors along with dense.
> >
> >Also, see if this makes sense, we are leaning towards commiting these
> >scala mappings in the current Mahout's trunk :[1]
> >
> >[1]
> >
> http://weatheringthrutechdays.blogspot.com/2013/07/scala-dsl-for-mahout-in
> >-core-linear.html
> >
> >-Dmitriy
> >
> >>
> >> --
> >> Reynold Xin, AMPLab, UC Berkeley
> >> http://rxin.org
> >>
> >>
> >>
> >> On Sat, Sep 7, 2013 at 6:13 AM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> >>wrote:
> >>
> >>> keep forgetting this: what is graphx release roadmap?
> >>>
> >>> On Fri, Sep 6, 2013 at 3:04 PM, Konstantin Boudnik <cos@apache.org>
> >>>wrote:
> >>> > Would it be more logical to use GraphX ?
> >>> >   https://amplab.cs.berkeley.edu/publication/graphx-grades/
> >>> >
> >>> > Cos
> >>> >
> >>> > On Fri, Sep 06, 2013 at 09:13PM, Mattmann, Chris A (398J) wrote:
> >>> >> Thanks Roman, I was thinking Giraph too (knew it supported graphs
> >>>but
> >>> >> wasn't sure it supported matrices). If Giraph supports matrices,
> >>>big +1.
> >>> >>
> >>> >> Cheers,
> >>> >> Chris
> >>> >>
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >> Chris Mattmann, Ph.D.
> >>> >> Senior Computer Scientist
> >>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> >> Office: 171-266B, Mailstop: 171-246
> >>> >> Email: chris.a.mattmann@nasa.gov
> >>> >> WWW:  http://sunset.usc.edu/~mattmann/
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >> Adjunct Assistant Professor, Computer Science Department
> >>> >> University of Southern California, Los Angeles, CA 90089 USA
> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> -----Original Message-----
> >>> >> From: Roman Shaposhnik <rvs@apache.org>
> >>> >> Date: Friday, September 6, 2013 2:00 PM
> >>> >> To: <dev@spark.incubator.apache.org>
> >>> >> Cc: "dev@sis.apache.org" <dev@sis.apache.org>
> >>> >> Subject: Re: Needs a matrix library
> >>> >>
> >>> >> >On Fri, Sep 6, 2013 at 1:33 PM, Mattmann, Chris A (398J)
> >>> >> ><chris.a.mattmann@jpl.nasa.gov> wrote:
> >>> >> >> Hey Martin,
> >>> >> >>
> >>> >> >> We may seriously consider using either Apache Hama here
(which
> >>>will
> >>> >> >> bring in Hadoop):
> >>> >> >
> >>> >> >On that note I'd highly recommend taking a look at Apache Giraph
> >>> >> >as well: http://giraph.apache.org/
> >>> >> >
> >>> >> >Thanks,
> >>> >> >Roman.
> >>> >> >
> >>> >>
> >>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message