mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hank Li <hank.h...@gmail.com>
Subject Re: save Mahout result in the memcached
Date Thu, 28 Oct 2010 00:55:46 GMT
Ted is correct, I am trying to use a cluster of machines to handle large
amount of recommendation requests. One design is to use hadoop to process
the raw data, then copy the result into the memcached or memcachedb. Another
design is to use memcached/b directly to process the data since it provides
unlimited memory for it to use. The key part is to let the recommendation
services to use memcached/b to do recommendation with fast response time.

Both memcached and memcachedb are good options, memcachedb is a little
complicated to setup than memcached though. failover issue can be solved by
other solutions too.

If anybody had done it or is going to test it, please share your experience.

Hank



On Mon, Oct 25, 2010 at 3:23 PM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Stuffing more memory and using a larger heap is definitely better on a
> single machine.
>
> Memcached would allow you to centralize that memory if you have a cluster
> of
> recommenders
> and with the right clever settings would probably let you have a limited
> kind of failure tolerance.
>
> I think that the confusion here might be that memcached is memory backed
> while memcachedb
> is the equivalent (almost) with persistence.  That single letter difference
> is very hard to see.
>
> Again, this is only interesting if you have a cluster of recommendation
> machines.  If you don't,
> Sean is exactly right that keeping the data on a single large machine is
> better.
>
> On Mon, Oct 25, 2010 at 3:00 PM, Sean Owen <srowen@gmail.com> wrote:
>
> > Perhaps I'm naive but wouldn't virtual memory be a better way to expand
> > memory by using storage? Or, I think you'd have to know there were
> > particular access patterns, and design your use of memcached to exploit
> > those well, to do better than swap.
> >
> > On Mon, Oct 25, 2010 at 10:26 PM, Hank Li <hank.h.li@gmail.com> wrote:
> >
> > > when the result is saved in the memory, mahout is running  very fast.
> > > memcached can expand the available memory, does anybody try to use
> > > memcached
> > > in mahout? what the pros and cons?
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message