spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liquan Pei <liquan...@gmail.com>
Subject Re: Creating a feature vector from text before using with MLLib
Date Wed, 01 Oct 2014 21:54:37 GMT
The program computes hashing bi-gram frequency normalized by total number
of bigrams then filter out zero values. hashing is a effective trick of
vectorizing features. Take a look at
http://en.wikipedia.org/wiki/Feature_hashing

Liquan

On Wed, Oct 1, 2014 at 2:18 PM, Soumya Simanta <soumya.simanta@gmail.com>
wrote:

> I'm trying to understand the intuition behind the features method that
> Aaron used in one of his demos. I believe this feature will just work for
> detecting the character set (i.e., language used).
>
> Can someone help ?
>
>
> def featurize(s: String): Vector = {
>   val n = 1000
>   val result = new Array[Double](n)
>   val bigrams = s.sliding(2).toArray
>
>   for (h <- bigrams.map(_.hashCode % n)) {
>     result(h) += 1.0 / bigrams.length
>   }
>
>   Vectors.sparse(n, result.zipWithIndex.filter(_._1 != 0).map(_.swap))
> }
>
>
>
>


-- 
Liquan Pei
Department of Physics
University of Massachusetts Amherst

Mime
View raw message