tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ken Krugler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TIKA-354) ProfilingHandler should take a length-limiting parameter
Date Sun, 01 Mar 2015 23:03:04 GMT

    [ https://issues.apache.org/jira/browse/TIKA-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342529#comment-14342529
] 

Ken Krugler commented on TIKA-354:
----------------------------------

Better speed is still important, as a 2x improvement from TIKA-1549 is good but means that
now it's only 45% of the web crawl time that's spent determining the language, versus 90%.
However the right way to do this is (with a new detector library) internally sampling until
the target confidence is reached, versus the caller having to decide how much text to analyze.

So net-net, yes I think this can be closed.

> ProfilingHandler should take a length-limiting parameter
> --------------------------------------------------------
>
>                 Key: TIKA-354
>                 URL: https://issues.apache.org/jira/browse/TIKA-354
>             Project: Tika
>          Issue Type: Improvement
>          Components: languageidentifier
>    Affects Versions: 0.5
>            Reporter: Vivek Magotra
>            Assignee: Ken Krugler
>         Attachments: TIKA-354-2.patch, TIKA-354.patch
>
>
> ProfilingHandler currently parses the entire document (thereby analyzing n-grams for
the entire doc).
> ProfilingHandler should take a length-limiting parameter that allows a user to specify
the amount of data that should get analyzed.
> In fact, by default that limit should be set to something like 8K.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message