mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edith Au <edith...@gmail.com>
Subject Re: recommendation based on user preference
Date Fri, 11 Jul 2014 15:11:37 GMT
Thanks for the suggestion Pat.  With the search engine approach, I would
imagine I could do ...

neighborhoodID<tab>bookstore,30 cafe,50 pet-store,10 pet-park,3

and then I can do a "LIKE" query to pick up the right docID and parse the
doc for weights.  But once I get a collection of neighborhoodID and
weights, I still need some way to compare similarity (between the user
existing neighborhood and the search results).  Now I am back to Mahout (or
some other math package I can use to find the strength)?

Thanks for the book recommendation.  I finished "Mahout in Action" last
week but I am sure the book is pretty out of date by now.


On Thu, Jul 10, 2014 at 8:01 PM, Pat Ferrel <pat.ferrel@gmail.com> wrote:

> You need hadoop installed on the machine you run on but don’t need HDFS or
> a cluster. This is called local mode where you set MAHOUT_LOCAL=true and
> use the local file system.
>
> If you want to customize the query at runtime I suggest a search engine.
> Using rowsimilarity you can only train in batch and can only pre-calculate
> recs. If you index the neighborhoods by feature you can construct a query
> at runtime and get fast results. So you can say that the user want pets
> (even though their current property doesn’t allow them). This would be
> something new, not related to their neighborhood. It is easily added to a
> search query. Weights are not so easy using a search engine but may not
> matter. Imagine indexing
>
> hieghborhoodID<tab>bookstore cafe pet-store pet-park
>
> heighborhoodID is the docID, the rest is the document (space delimited set
> of tokens). Index this with Solr or something else. Then at runtime do a
> fulltext query with “bookstore pet-store pet-park” or however you want to
> build the query. This is actually the way we are thinking about the next
> gen of recommender—using a search engine even for collaborative filtering
> data.
>
> Look for a book by Ted Dunning called “Practical Machine Learning”, which
> talks about this approach.
>
> On Jul 10, 2014, at 5:07 PM, Edith Au <edith.au@gmail.com> wrote:
>
> I was under the impression that I can only run RowSimilarityJob with
> Hadoop.  I will take a look at that.  Thx!
>
> I see your point about retraining the data over and over.  But there are
> couple other requirements I left out from my original post.
>
> 1. There are amenities in a user existing neighborhood where she does not
> care for.  For example, if she does not have any children, schools and
> day-care centers in her existing neighborhood (or suggesting neighborhoods)
> should not sway the final score.
>
> 2. My next feature is to inject customization.  For example, she may want
> to move to a pet friendly area because her current neighborhood does not
> have enough facilities (eg. Vets, pets store, pets friendly parks) for her
> pet.
>
> If I pre-calculate the row matrix of similar neighborhoods, I am not sure
> how I can implement the customization (by adding or removing amenities
> requirements at runtime).    Any thought on that?
>
> Thanks for the reminder on mahout FastID.  It could easily be a newbie
> mistake to use a regular int or long for mapping.
>
> Thanks again for your help.  Much appreciated!
>
>
>
>
> On Thu, Jul 10, 2014 at 1:40 PM, Pat Ferrel <pat.ferrel@gmail.com> wrote:
>
> > Doing things this way you are using the neighborhoodID as a proxy for
> > userID/rowID in the recommender. I don’t see the benefit of the in-memory
> > version here since all output can easily be pre-calculated. Then it will
> > only be a lookup at runtime. You can use “rowsimilarity" on a single
> > machine without setting up a cluster, just use the local filesystem. This
> > is the way I’d do it.
> >
> > You definitely don’t want to “only load the columns (amenities) correlate
> > to a selected user” with an in-memory recommender. This loading of data
> > will trigger a retraining of the recommender before you can ask it for
> > similar neighborhoods and that will take more time than you want. This
> > potentially will happen as each new user visits your app.
> >
> > However if you train on all amenities for all neighborhoods then the
> > in-memory recommender should work and would train only once. Your data
> > would look like: (heighborhoodID, cafesID, numberOfCafes) and so on for
> > every non-zero cell in the table. And remember that ALL IDs must be
> Mahout
> > IDs—you can’t use your own IDs. Mahout IDs correspond to matrix
> > coordinants, they are ordinal Ints. Think of them as the row and column
> > number of the table.
> >
> > On Jul 10, 2014, at 10:45 AM, Edith Au <edith.au@gmail.com> wrote:
> >
> > Thank you so much for the suggestions.  It took me sometime to figure
> > things out but I believe I have a pretty good grip on what's need to be
> > done now. My dataset is small enough to fit into a single machine so I am
> > going to use an in memory implementation rather than hadoop.   As
> suggested
> > by both Pat and Manuel, I have a table (in file system) with
> neighborhoods
> > as rows and amenities as columns.  In runtime, I will only load the
> columns
> > (amenities) correlate to a selected user and do a UserSimilarity
> operation
> > between each neighborhood and the one the user resides in.  After that, I
> > can pick up the NearestNUserNeighborhoods for results.
> >
> > I gather UserSimilarity is the in-memory equivalent of RowSimilarity
> > (Hadoop) ?  It would be great if someone can confirm it!
> >
> > Thanks again Pat and Manuel!
> > Edith
> >
> >
> > On Wed, Jul 2, 2014 at 4:06 PM, Pat Ferrel <pat.ferrel@gmail.com> wrote:
> >
> >> If you are looking to recommend a similar neighborhood based on the
> >> characteristics of some other neighborhood (the user’s current one) so
> > you
> >> wouldn’t use collaborative filtering. This is a metadata recommender
> > based
> >> on similarity of neighborhoods not a collection of user preferences.
> >>
> >> The easiest and fastest would be to use a search engine but I’ll leave
> >> that for now since it doesn’t account for feature weights as well.
> >>
> >> create a table like this:
> >> Neighborhood    Gym Cafe        Bookstore
> >> Downtown        15      50              0
> >> Midtown         30      100             10
> >> …
> >>
> >> You will need to convert the row IDs into sequential ints, which Mahout
> >> uses for IDs. Then read them into a sequenceFile creating a Distributed
> > Row
> >> Matrix, which has Key -  Value pairs. Keys = the integer neighborhood
> > IDs,
> >> the Value is a Vector (a sort of list) of column integer IDs with the
> >> counts.
> >>
> >> Then run rowsimilarity on the DRM. This is the CLI but there is also a
> >> Driver you can call from your code.
> >>
> >> There are some data prep issues you will have since larger neighborhoods
> >> will have higher counts. An easy thing to do would be to normalize the
> >> counts by something like population or physical size so you get cafes
> per
> >> resident or per sq mile or some other ratio.
> >>
> >> The result of the rowsimilarity job will be another DRM of key =
> >> neightborhood ID, values = Vector of similar neighborhoods (by integer
> > ID)
> >> with a strength of similarity. Sort the vector by strength and you’ll
> > have
> >> an ordered list of similar neighborhoods for each neighborhood.
> >>
> >> On Jun 30, 2014, at 12:48 PM, Edith Au <edith.au@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >>
> >> I am a newbie and am looking for some guidance to implement my
> >> recommender.  Any help would be greatly appreciated.  I have a small
> >> data set of location information with the following fields:
> >> neighborhood, amenities, and counts.  For example:
> >>
> >> Downtown          Gym 15
> >> Downtown          Cafe 50
> >> …
> >> Midtown             Gym 30
> >> Midtown             Cafe 100
> >> Midtown             Bookstore 10
> >> ...
> >> Financial Dist
> >> …
> >>
> >>
> >> so on and so forth.  I want to recommend a neighborhood for a user to
> >> reside base on the amenities (and some other metrics) in his/her
> >> current neighborhood.    My understanding is that model-based
> >> recommendation would be a good fit for the job.  If I am on the right
> >> track,  is there a experimental/beta recommender I can try?
> >>
> >>
> >> If there is no such recommender yet, can I still use Mahout for my
> >> project?  For example, can I implement my own Similarity which only
> >> computes the similarity between one user's preference to a set of
> >> neighborhood?  If I understand Mahout correctly, User/Item Similarity
> >> would do N x (N-1) pair of comparisons as oppose to 1 x N comparisons.
> >> In my example, User/Item Similarity would compare between Downtown,
> >> Midtown, Fin Dist -- which would be a waste in computation resources
> >> since the comparisons are not needed.
> >>
> >>
> >> Thanks in advance for your help.
> >>
> >> Edith
> >>
> >>
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message