tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ken Krugler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TIKA-369) Improve accuracy of language detection
Date Thu, 07 Feb 2013 17:01:15 GMT

    [ https://issues.apache.org/jira/browse/TIKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573670#comment-13573670
] 

Ken Krugler commented on TIKA-369:
----------------------------------

I've been using language-detection in another project for six months. In general it works
better than what's in Tika, but has a number of design/coding issues (gnarly singleton DetectorFactory,
assumption that profiles are loaded from external files, problems with setting a priori language
probabilities). I've got a fork of it with some fixes, but it's not ready for prime time.

So net-net is a mild +1 from me, but I think there may be some post-integration challenges.
                
> Improve accuracy of language detection
> --------------------------------------
>
>                 Key: TIKA-369
>                 URL: https://issues.apache.org/jira/browse/TIKA-369
>             Project: Tika
>          Issue Type: Improvement
>          Components: languageidentifier
>    Affects Versions: 0.6
>            Reporter: Ken Krugler
>            Assignee: Ken Krugler
>         Attachments: lingdet-mccs.pdf, Surprise and Coincidence.pdf, textcat.pdf
>
>
> Currently the LanguageProfile code uses 3-grams to find the best language profile using
Pearson's chi-square test. This has three issues:
> 1. The results aren't very good for short runs of text. Ted Dunning's paper (attached)
indicates that a log-likelihood ratio (LLR) test works much better, which would then make
language detection faster due to less text needing to be processed.
> 2. The current LanguageIdentifier.isReasonablyCertain() method uses an exact value as
a threshold for certainty. This is very sensitive to the amount of text being processed, and
thus gives false negative results for short runs of text.
> 3. Certainty should also be based on how much better the result is for language X, compared
to the next best language. If two languages both had identical sum-of-squares values, and
this value was below the threshold, then the result is still not very certain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message