spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sameer Tilak <ssti...@live.com>
Subject RE: MLLib decision tree: Weights
Date Wed, 03 Sep 2014 17:21:10 GMT
Dear Xiangrui,
Thanks for your reply. We will use sampling for now. However, just to let you know, we believe
that it is not the best fit for our problems due to two reasons (1) high dimensionality of
data (600) features and (2) Highly skewed distribution. 

Do you have any idea when MLLib v1.2 will be released? We can plan things accordingly.
> Date: Tue, 2 Sep 2014 23:15:09 -0700
> Subject: Re: MLLib decision tree: Weights
> From: mengxr@gmail.com
> To: sstilak@live.com
> CC: user@spark.apache.org
> 
> This is not supported in MLlib. Hopefully, we will add support for
> weighted examples in v1.2. If you want to train weighted instances
> with the current tree implementation, please try importance sampling
> first to adjust the weights. For instance, an example with weight 0.3
> is sampled with probability 0.3. And if it is sampled, its weight
> become 1. -Xiangrui
> 
> On Tue, Sep 2, 2014 at 1:05 PM, Sameer Tilak <sstilak@live.com> wrote:
> > Hi everyone,
> >
> >
> > We are looking to apply a weight to each training example; this weight
> > should be used when computing the penalty of a misclassified example.  For
> > instance, without weighting, each example is penalized 1 point when
> > evaluating the model of a classifier, such as a decision tree.  We would
> > like to customize this penalty for each training example, such that we could
> > apply a penalty of W for a misclassified example, where W is a weight
> > associated with the given training example.
> >
> >
> > Is this something that is supported directly in MLLib? I would appreciate if
> > someone can point me in right direction.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 
 		 	   		  
Mime
View raw message