spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peng Meng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-18088) ChiSqSelector FPR PR cleanups
Date Tue, 25 Oct 2016 12:38:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605211#comment-15605211
] 

Peng Meng commented on SPARK-18088:
-----------------------------------

Hi [~josephkb] , I am not quite understand "Testing against only the p-value and not the test
statistic does not really tell you anything. " SelectFPR in sklearn is only based on pValue.


> ChiSqSelector FPR PR cleanups
> -----------------------------
>
>                 Key: SPARK-18088
>                 URL: https://issues.apache.org/jira/browse/SPARK-18088
>             Project: Spark
>          Issue Type: Bug
>          Components: ML
>            Reporter: Joseph K. Bradley
>            Assignee: Joseph K. Bradley
>
> There are several cleanups I'd like to make as a follow-up to the PRs from [SPARK-17017]:
> * Rename selectorType values to match corresponding Params
> * Add Since tags where missing
> * a few minor cleanups
> One major item: FPR is not implemented correctly.  Testing against only the p-value and
not the test statistic does not really tell you anything.  We should follow sklearn, which
allows a p-value threshold for any selection method: [http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFpr.html]
> * In this PR, I'm just going to remove FPR completely.  We can add it back in a follow-up
PR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message