spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iman Mohtashemi <iman.mohtash...@gmail.com>
Subject Re: TallSkinnyQR
Date Tue, 08 Nov 2016 14:49:23 GMT
So
 b =
0.89
0.42
0.0
0.88
0.97
The solution at the bottom is the solution to Ax = b solved using Gaussian
elimination. I guess another question is, is there another way to solve
this problem? I'm trying to solve the least squares fit with a huge A (5MM
x 1MM)

x = inverse(A-transpose*A)*A-transose*b

but I didn't see any functions for matrix inversion

I suppose I can use an iterative solver but I didn't see that either which
is why I chose the QR decomposition , solve for Q and then Q-transpose*b =
d and the solve Lx = d which would give the solution. But I don't think
this would work either since the matrices are local copies and not RDD data
structures. Any advice would be appreciated...
Iman

P.S. I also looked in the linear regression class in the mlib but I haven't
seen any examples with sparse matrix and sparse vectors as the input just
'Dataset' If you have a code example of this this would work??


On Tue, Nov 8, 2016 at 6:41 AM Iman Mohtashemi <iman.mohtashemi@gmail.com>
wrote:

> Hi Sean,
> Here you go:
>
> sparsematrix.txt =
>
> row, col ,val
> 0,0,.42
> 0,1,.28
> 0,2,.89
> 1,0,.83
> 1,1,.34
> 1,2,.42
> 2,0,.23
> 3,0,.42
> 3,1,.98
> 3,2,.88
> 4,0,.23
> 4,1,.36
> 4,2,.97
>
> The vector is just the third column of the matrix which should give the
> trivial solution of [0,0,1]
>
> This translates to this which is correct
> There are zeros in the matrix (Not really sparse but just an example)
> 0.42  0.28  0.89
> 0.83  0.34  0.42
> 0.23  0.0   0.0
> 0.42  0.98  0.88
> 0.23  0.36  0.97
>
>
> Here is what I get for  the Q and R
>
> Q: -0.21470961288429483  0.23590615093828807   0.6784910613691661
> -0.3920784235278427   -0.06171221388256143  0.5847874866876442
> -0.7748216464954987   -0.4003560542230838   -0.29392323671555354
> -0.3920784235278427   0.8517909521421976    -0.31435038559403217
> -0.21470961288429483  -0.23389547730301666  -0.11165321782745863
> R: -1.0712142642814275  -0.8347536340918976  -1.227672225670157
> 0.0                  0.7662808691141717   0.7553315911660984
> 0.0                  0.0                  0.7785210939368136
>
> When running this in matlab the numbers are the same but row 1 is the last
> row and the last row is interchanged with row 3
>
>
>
> On Mon, Nov 7, 2016 at 11:35 PM Sean Owen <sowen@cloudera.com> wrote:
>
> Rather than post a large section of code, please post a small example of
> the input matrix and its decomposition, to illustrate what you're saying is
> out of order.
>
> On Tue, Nov 8, 2016 at 3:50 AM im281 <iman.mohtashemi@gmail.com> wrote:
>
> I am getting the correct rows but they are out of order. Is this a bug or
> am
> I doing something wrong?
>
>
>

Mime
View raw message