spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nishanth P S <nishant...@gmail.com>
Subject Re: How to use BigInteger for userId and productId in collaborative Filtering?
Date Thu, 15 Jan 2015 06:49:10 GMT
Hi Xiangrui,

Thanks! for the reply, I will explore the suggested solutions.

-Nishanth
Hi Nishanth,

Just found out where you work:) We had some discussion in
https://issues.apache.org/jira/browse/SPARK-2465 . Having long IDs
will increase the communication cost, which may not worth the benefit.
Not many companies have more than 1 billion users. If they do, maybe
they can mirror the implementation for their use cases. I can suggest
several possible solutions:

1. Hash user IDs into integers before training. If the collision rate
is high and it is crucial for your business, you can recompute user
features from product features by solving least squares after
training. This works when the product IDs could be mapped to integers.
2. Make type aliases in ALS, so that you can easily mirror the
implementation to use long IDs and track future changes.
3. Make ALS implementation use generic ID types. This would be the
best solution, but it requires some refactoring of the code.

Best,
Xiangrui

On Wed, Jan 14, 2015 at 1:04 PM, Nishanth P S <nishanthps@gmail.com> wrote:
> Yes, we are close to having more 2 billion users. In this case what is the
> best way to handle this.
>
> Thanks,
> Nishanth
>
> On Fri, Jan 9, 2015 at 9:50 PM, Xiangrui Meng <mengxr@gmail.com> wrote:
>>
>> Do you have more than 2 billion users/products? If not, you can pair
>> each user/product id with an integer (check RDD.zipWithUniqueId), use
>> them in ALS, and then join the original bigInt IDs back after
>> training. -Xiangrui
>>
>> On Fri, Jan 9, 2015 at 5:12 PM, nishanthps <nishanthps@gmail.com> wrote:
>> > Hi,
>> >
>> > The userId's and productId's in my data are bigInts, what is the best
>> > way to
>> > run collaborative filtering on this data. Should I modify MLlib's
>> > implementation to support more types? or is there an easy way.
>> >
>> > Thanks!,
>> > Nishanth
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-use-BigInteger-for-userId-and-productId-in-collaborative-Filtering-tp21072.html
>> > Sent from the Apache Spark User List mailing list archive at
Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>
>

Mime
View raw message