mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Detecting high bias and variance in AdaptiveLogisticRegression classification
Date Tue, 03 Dec 2013 03:50:50 GMT
Inline


On Mon, Dec 2, 2013 at 8:55 AM, optimusfan <optimusfan@yahoo.com> wrote:

> ... To accomplish this, we used AdaptiveLogisticRegression and trained 46
> binary classification models.  Our approach has been to do an 80/20 split
> on the data, holding the 20% back for cross-validation of the models we
> generate.
>

Sounds reasonable.


> We've been playing around with a number of different parameters, feature
> selection, etc. and are able to achieve pretty good results in
> cross-validation.


When you say cross validation, do you mean the magic cross validation that
the ALR uses?  Or do you mean your 20%?


>  We have a ton of different metrics we're tracking on the results, most
> significant to this discussion is that it looks like we're achieving very
> good precision (typically >.85 or .9) and a good f1-score (typically again
> >.85 or .9).


These are extremely good results.   In fact they are good enough I would
starting thinking about a target leak.

 However, when we then take the models generated and try to apply them to
> some new documents, we're getting many more false positives than we would
> expect.  Documents that should have 2 categories are testing positive for
> 16, which is well above what I'd expect.  By my math I should expect 2 true
> positives, plus maybe 4.4 (.10 false positives * 44 classes) additional
> false positives.
>

You said documents.  Where do these documents come from?

One way to get results just like you describe is if you train on raw news
wire that is split randomly between training and test.  What can happen is
that stories that get edited and republished have a high chance of getting
at least one version in both training and test.  This means that the
supposedly independent test set actually has significant overlap with the
training set.  If your classifier over-fits, then the test set doesn't
catch the problem.

Another way to get this sort of problem is if you do your training/test
randomly, but the new documents come from a later time.  If your classifier
is a good classifier, but is highly specific to documents from a particular
moment in time, then your test performance will be a realistic estimate of
performance for contemporaneous documents but will be much higher than
performance on documents from a later point in time.

A third option could happen if your training and test sets were somehow
scrubbed of poorly structured and invalid documents.  This often happens.
 Then, in the real system, if the scrubbing is not done, the classifier may
fail because the new documents are not scrubbed in the same way as the
training documents.

These are just a few of the ways that *I* have screwed up building
classifiers.  I am sure that there are more.

We suspected that perhaps our models were underfitting or overfitting,
> hence this post.  However, I'll take any and all suggestions for anything
> else we should be looking at.
>

Well, I think that, almost by definition, you have an overfitting problem
of some kind.  The question is what kind.  The only think that I think that
you don't have is a frank target leak in your documents.  That would
(probably) have given you even higher scores on your test case.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message