mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: is it possible to compute the SVD for a large scale matrix
Date Fri, 25 Mar 2011 16:56:39 GMT
Please supply a bit more information.

First, how many average non-zero elements per row do you have?

Secondly, how many non-zero values is in the most full row?

How did you configure your map-reduce program?

What kind of machines were you running on?

On Fri, Mar 25, 2011 at 3:25 AM, Wei Li <wei.lee04@gmail.com> wrote:

> Actually, I would like to perform the spectral clustering on a large scale
> sparse matrix, but it failed due to the OutOfMemory error when creating the
> DenseMatrix for SVD decomposition.
>
> Best
> Wei
>
> On Fri, Mar 25, 2011 at 4:05 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> wrote:
>
> > SSVD != Lanczos. if you do PCA or LSI it is perhaps what you need. it
> > can take on these things. Well at least some of my branches can, if
> > not the official patch.
> >
> > -d
> >
> > On Thu, Mar 24, 2011 at 11:09 PM, Wei Li <wei.lee04@gmail.com> wrote:
> > >
> > > thanks for your reply
> > >
> > > my matrix is not very dense, a sparse matrix.
> > >
> > > I have tried the svd of Mahout, but failed due to the OutOfMemory
> error.
> > >
> > > Best
> > > Wei
> > >
> > >
> > >
> > > On Fri, Mar 25, 2011 at 2:03 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> > wrote:
> > >>
> > >> you can certainly try to write it out into a DRM (distributed row
> > >> matrix) and run stochastic SVD on  hadoop (off the trunk now). see
> > >> MAHOUT-593. This is suitable if you have a good decay of singular
> > >> values (but if you don't it probably just means you have so much noise
> > >> that it masks the problem you are trying to solve in your data).
> > >>
> > >> Current committed solution is not most efficient yet, but it should be
> > >> quite capable.
> > >>
> > >> If you do, let me know how it went.
> > >>
> > >> thanks.
> > >> -d
> > >>
> > >> On Thu, Mar 24, 2011 at 10:59 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
> >
> > >> wrote:
> > >> > Are you sure your matrix is dense?
> > >> >
> > >> > On Thu, Mar 24, 2011 at 9:59 PM, Wei Li <wei.lee04@gmail.com>
> wrote:
> > >> >> Hi All:
> > >> >>
> > >> >>    is it possible to compute the SVD factorization for a 600,000
*
> > >> >> 600,000
> > >> >> matrix using Mahout?
> > >> >>
> > >> >>    I have got the OutOfMemory error when creating the DenseMatrix.
> > >> >>
> > >> >> Best
> > >> >> Wei
> > >> >>
> > >> >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message