lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Harwood (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-6747) FingerprintFilter - a TokenFilter for clustering/linking purposes
Date Fri, 21 Aug 2015 14:57:46 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mark Harwood updated LUCENE-6747:
---------------------------------
    Attachment: fingerprintv3.patch

Updated patch - removed instanceof check and added entry to Changes.txt.

Will commit to trunk and 5.x in a day or two if there's no objections

> FingerprintFilter - a TokenFilter for clustering/linking purposes
> -----------------------------------------------------------------
>
>                 Key: LUCENE-6747
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6747
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: modules/analysis
>            Reporter: Mark Harwood
>            Priority: Minor
>         Attachments: fingerprintv1.patch, fingerprintv2.patch, fingerprintv3.patch
>
>
> A TokenFilter that emits a single token which is a sorted, de-duplicated set of the input
tokens.
> This approach to normalizing text is used in tools like OpenRefine[1] and elsewhere [2]
to help in clustering or linking texts.
> The implementation proposed here has a an upper limit on the size of the combined token
which is output.
> [1] https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth
> [2] https://rajmak.wordpress.com/2013/04/27/clustering-text-map-reduce-in-python/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message