tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Allison (Jira)" <j...@apache.org>
Subject [jira] [Created] (TIKA-2937) Improve legacy HTML charset detector by replicating Standard's behavior for UTF-16
Date Fri, 06 Sep 2019 14:13:00 GMT
Tim Allison created TIKA-2937:
---------------------------------

             Summary: Improve legacy HTML charset detector by replicating Standard's behavior
for UTF-16
                 Key: TIKA-2937
                 URL: https://issues.apache.org/jira/browse/TIKA-2937
             Project: Tika
          Issue Type: Improvement
            Reporter: Tim Allison


Our newer standards based detector follows the standard and automatically modifies a charset
header of "UTF-16" (and some variants) to "UTF-8".  This appears to help quite a bit.

||LegacyHTMLDetector||StandardHTMLDetector||NumCommonTokensLegacy||NumCommonTokensStandard||Delta||
|UTF-16LE|UTF-8|10|27351|27341|
|UTF-16BE|UTF-8|14|43565|43551|
|UTF-16|UTF-8|6099|748024|741925|



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message