spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shahab <shahab.mok...@gmail.com>
Subject Re: Is there any Sparse Matrix implementation in Spark/MLib?
Date Sun, 01 Mar 2015 10:55:27 GMT
Thanks Vijay, but the setup requirement for GML was not straightforward for
me at all, so I put it aside for a while.

best,
/Shahab

On Sun, Mar 1, 2015 at 9:34 AM, Vijay Saraswat <vijay@saraswat.org> wrote:

>  GML is a fast, distributed, in-memory sparse (and dense) matrix
> libraries.
>
> It does not use RDDs for resilience. Instead we have examples that use
> Resilient X10 (which provides recovery of distributed control structures in
> case of node failure) and Hazelcast for stable storage.
>
> We are looking to benchmark with RDDs to compare overhead, and also
> looking to see how the same ideas could be realized on top of RDDs.
>
>
>
> On 2/28/15 7:25 PM, Joseph Bradley wrote:
>
> Hi Shahab,
>
>  There are actually a few distributed Matrix types which support sparse
> representations: RowMatrix, IndexedRowMatrix, and CoordinateMatrix.
> The documentation has a bit more info about the various uses:
> http://spark.apache.org/docs/latest/mllib-data-types.html#distributed-matrix
>
>  The Spark 1.3 RC includes a new one: BlockMatrix.
>
>  But since these are distributed, they are represented using RDDs, so
> they of course will not be as fast as computations on smaller, locally
> stored matrices.
>
>  Joseph
>
> On Fri, Feb 27, 2015 at 4:39 AM, Ritesh Kumar Singh <
> riteshoneinamillion@gmail.com> wrote:
>
>> try using breeze (scala linear algebra library)
>>
>> On Fri, Feb 27, 2015 at 5:56 PM, shahab <shahab.mokari@gmail.com> wrote:
>>
>>> Thanks a lot Vijay, let me see how it performs.
>>>
>>>  Best
>>> Shahab
>>>
>>>
>>> On Friday, February 27, 2015, Vijay Saraswat <vijay@saraswat.org> wrote:
>>>
>>>> Available in GML --
>>>>
>>>>
>>>> http://x10-lang.org/x10-community/applications/global-matrix-library.html
>>>>
>>>> We are exploring how to make it available within Spark. Any ideas would
>>>> be much appreciated.
>>>>
>>>> On 2/27/15 7:01 AM, shahab wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I just wonder if there is any Sparse Matrix implementation available
>>>>> in Spark, so it can be used in spark application?
>>>>>
>>>>> best,
>>>>> /Shahab
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>
>
>

Mime
View raw message