spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert_dodier <>
Subject Re: How to handle categorical variables in Spark MLlib?
Date Sat, 26 Dec 2015 22:53:15 GMT
hokam chauhan wrote
> So how the string value of categorical variable can be converted into
> double values for forming the features vector ?

Well, the key characteristic of the variables is that their values are not
ordered. So the representation you choose has to honor that. If the model is
doing some arithmetic on the inputs (e.g. a logistic regression model
computes a weighted sum of the inputs) or otherwise assuming an ordering of
values, then the appropriate representation is the so-called "one hot"
representation, in which a categorical variable of n possible values is
represented as a vector of length n, in which exactly one element is 1 and
the rest are 0.

Depending on the models you are using, other representations might be
possible. But a one-hot representation is widely applicable.

> Also how the weight for individual categories can be calculated for
> models. Like we have Gender as variable with categories as Male and Female
> and we want to give more weight to female category, then how this can be
> accomplished?  

Well, it probably depends on exactly what you mean by "more weight". If you
mean that one category is under-represented in the available data, and you
want to assume, let's say, that each datum in that category ought to count
the same as two data in another category, you could just create a data set
with an extra copy of those data. An equivalent method is to allow for
weighting the log-likelihood or other goodness of fit function. That's more
convenient and flexible (it allows for noninteger weights), but I don't
remember if Spark supports that. 

If you mean some other kind of weighting, you'll have to explain more about
what you're trying to achieve.

> Also is there a way through which string values from raw text can be
> converted to features vector(Apart from the HashingTF-IDF transformation)
> ?

I don't know any other method. Maybe someone else can suggest something.


Robert Dodier

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message