spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject OOM when making bins in BinaryClassificationMetrics ?
Date Sun, 02 Nov 2014 17:34:49 GMT
This might be a question for Xiangrui. Recently I was using
BinaryClassificationMetrics to build an AUC curve for a classifier
over a reasonably large number of points (~12M). The scores were all
probabilities, so tended to be almost entirely unique.

The computation does some operations by key, and this ran out of
memory. It's something you can solve with more than the default amount
of memory, but in this case, it seemed unuseful to create an AUC curve
with such fine-grained resolution.

I ended up just binning the scores so there were ~1000 unique values
and then it was fine.

Does that sound generally useful as some kind of parameter? or am I
missing a trick here.

Sean

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message