tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ken Krugler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (TIKA-344) Charset hint in metadata
Date Sat, 12 Dec 2009 22:23:18 GMT

    [ https://issues.apache.org/jira/browse/TIKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789806#action_12789806
] 

Ken Krugler commented on TIKA-344:
----------------------------------

It would be useful for various detectors of charset & language to be able to (a) use different
metadata keys for their results, and (b) include a confidence level. That way you could have
a top-level resolver that combined the results with all knowledge, including incoming hints,
to pick the best result.

Though note that for HTML pages, there's a patch to use the charset found in meta tags, which
is usually pretty good (and definitely better than the server response header charset or auto-detected
charset). See https://issues.apache.org/jira/browse/TIKA-332, as well as:

https://issues.apache.org/jira/browse/TIKA-333

https://issues.apache.org/jira/browse/TIKA-334

https://issues.apache.org/jira/browse/TIKA-335

https://issues.apache.org/jira/browse/TIKA-341

> Charset hint in metadata
> ------------------------
>
>                 Key: TIKA-344
>                 URL: https://issues.apache.org/jira/browse/TIKA-344
>             Project: Tika
>          Issue Type: Improvement
>          Components: parser
>    Affects Versions: 0.6
>            Reporter: Piotr B.
>            Priority: Minor
>
> It would be nice if TextParser and HtmlParser support Metadata.CONTENT_ENCODING hint.
> In my application I always prefer that hint (if it is present) over the charset detector
result, because charset detector is often wrong on short inputs (even if  match.confidence
is 100) and I know that hint if present is right in 99%.
> To be more general, user might be able to change default behaviour by override a function
 F(hint, detectorResults) -> charset. 
> Other solution is to create some standard strategies and let user to choose one of them:
> a) hint is most important
> b) charset detector result is most important
> c) create some heuristic using detectorResult.confidence, hint and maybe input length
> Maybe the last heuristic method would be good enough for most cases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message