spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debasish Das <>
Subject Re: Which linear algebra interface to use within Spark MLlib?
Date Thu, 19 Mar 2015 06:21:53 GMT
dgemm dgemv and dot come to Breeze and Spark through netlib-java....

Right now both in dot and dgemv Breeze does a extra memory allocate but we
already found the issue and we are working on adding a common trait that
will provide a sink operation (basically memory will be allocated by
user)...adding more BLAS operators in breeze will also help in general as
lot more operations are defined over there...

On Wed, Mar 18, 2015 at 8:09 PM, Ulanov, Alexander <>

> Hi,
> Currently I am using Breeze within Spark MLlib for linear algebra. I would
> like to reuse previously allocated matrices for storing the result of
> matrices multiplication, i.e. I need to use "gemm" function C:=q*A*B+p*C,
> which is missing in Breeze (Breeze automatically allocates a new matrix to
> store the result of multiplication). Also, I would like to minimize gemm
> calls that Breeze does. Should I use mllib.linalg.BLAS functions instead?
> While it has gemm and axpy, it has rather limited number of operations. For
> example, I need sum of the matrix by row or by columns, or applying a
> function to all elements in a matrix. Also, MLlib Vector and Matrix
> interfaces that linalg.BLAS operates seems to be rather undeveloped. Should
> I use plain netlib-java instead (will it remain in MLlib in future
> releases)?
> Best regards, Alexander

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message