tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Allison (Jira)" <j...@apache.org>
Subject [jira] [Updated] (TIKA-2937) Improve legacy HTML charset detector by replicating Standard's behavior for UTF-16
Date Fri, 06 Sep 2019 14:14:00 GMT

     [ https://issues.apache.org/jira/browse/TIKA-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tim Allison updated TIKA-2937:
------------------------------
    Priority: Minor  (was: Major)

> Improve legacy HTML charset detector by replicating Standard's behavior for UTF-16
> ----------------------------------------------------------------------------------
>
>                 Key: TIKA-2937
>                 URL: https://issues.apache.org/jira/browse/TIKA-2937
>             Project: Tika
>          Issue Type: Improvement
>            Reporter: Tim Allison
>            Priority: Minor
>
> Our newer standards based detector follows the standard and automatically modifies a
charset header of "UTF-16" (and some variants) to "UTF-8".  This appears to help quite a bit.
> ||LegacyHTMLDetector||StandardHTMLDetector||NumCommonTokensLegacy||NumCommonTokensStandard||Delta||
> |UTF-16LE|UTF-8|10|27351|27341|
> |UTF-16BE|UTF-8|14|43565|43551|
> |UTF-16|UTF-8|6099|748024|741925|



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message