spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yanwei Wayne Zhang <>
Subject Re: Invert large matrix
Date Thu, 29 Dec 2016 22:30:53 GMT
Thanks for the advice. I figured out a way to solve this problem by avoiding the matrix representation.


From: Sean Owen <>
Sent: Thursday, December 29, 2016 1:52 PM
To: Yanwei Wayne Zhang; user
Subject: Re: Invert large matrix

I think the best advice is: don't do that. If you're trying to solve a linear system, solve
the linear system without explicitly constructing a matrix inverse. Is that what you mean?

On Thu, Dec 29, 2016 at 2:22 AM Yanwei Wayne Zhang <<>>

Hi all,

I have a matrix X stored as RDD[SparseVector] that is high dimensional, say 800 million rows
and 2 million columns, and more 95% of the entries are zero.

Is there a way to invert (X'X + eye) efficiently, where X' is the transpose of X and eye is
the identity matrix? I am thinking of using RowMatrix but not sure if it is feasible.

Any suggestion is highly appreciated.



View raw message