tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andreas Meier (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (TIKA-2592) HTML with charset unicode handled as utf-16 instead utf-8
Date Mon, 05 Mar 2018 12:47:00 GMT

    [ https://issues.apache.org/jira/browse/TIKA-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386017#comment-16386017
] 

Andreas Meier edited comment on TIKA-2592 at 3/5/18 12:46 PM:
--------------------------------------------------------------

Thanks Tim, but I think I will just download the govdocs1 and work on my own system. (If I
find something special there, I will reference it)

Too many people and nightly builds stressing one vm may be too much.

 

I already have a small testset I run tika against (~300k+ files), that is also the reason
for the numerous tickets I created lately.

 

 
{quote}That you were referring to the difference between what Java's CharSet class says is
a valid name, and the set of names in the IANA standard, right?{quote}

Yes

{quote}If so, then I agree that what's needed is a list of valid IANA charset names, and we
should ignore any charset name in the HTML that's not in this set. Unfortunately Java appears
to have a bug, where even though "unicode" isn't a valid IANA charset name, that CharSet returns
true from the isRegistered() method. So to make this work, we'd need to roll our own, e.g.
using the attached list of names.{quote}

Thanks for the charset names upload.
I evaluated sun.nio.cs.StandardCharsets which is used to retrieve the java charsets based
on a given string.

So I tried to determine the strings that do not occur in the IANA list and therefore might
reference a wrong java charset.
The results are in https://issues.apache.org/jira/secure/attachment/12913030/StandardCharsets_unsupported_by_IANA.txt
(I used columns "Preferred MIME Name", "Name" and "Aliases")

Posted for completion and to create a testset with possibly unsupported charset-names.

(sorry for the bad english...)


was (Author: andreasmeier):
Thanks Tim, but I think I will just download the govdocs1 and work on my own system. (If I
find something special there, I will reference it)

Too many people and nightly builds stressing one vm may be too much.

 

I already have a small testset I run tika against (~300k+ files), that is also the reason
for the numerous tickets I created lately.

 

 
{quote}That you were referring to the difference between what Java's CharSet class says is
a valid name, and the set of names in the IANA standard, right?{quote}

Yes

{quote}If so, then I agree that what's needed is a list of valid IANA charset names, and we
should ignore any charset name in the HTML that's not in this set. Unfortunately Java appears
to have a bug, where even though "unicode" isn't a valid IANA charset name, that CharSet returns
true from the isRegistered() method. So to make this work, we'd need to roll our own, e.g.
using the attached list of names.{quote}

Thanks for the charset names upload.
I evaluated sun.nio.cs.StandardCharsets which is used to retrieve the java charsets based
on a given string.

So I tried to determine the strings that do not occur in the IANA list and therefore might
reference a wrong java charset.
The results are in https://issues.apache.org/jira/secure/attachment/12913030/StandardCharsets_unsupported_by_IANA.txt
(I used columns "Preferred MIME Name", "Name" and "Aliases")

Posted for completion.

(sorry for the bad english...)

> HTML with charset unicode handled as utf-16 instead utf-8
> ---------------------------------------------------------
>
>                 Key: TIKA-2592
>                 URL: https://issues.apache.org/jira/browse/TIKA-2592
>             Project: Tika
>          Issue Type: Improvement
>    Affects Versions: 2.0, 1.16, 1.17
>            Reporter: Andreas Meier
>            Priority: Minor
>         Attachments: IANA Charset names.txt, StandardCharsets_unsupported_by_IANA.txt,
TestCharsetUnicodeHTML.html, TestHTMLCharsetArabicCP1256.html, TestHTMLCharsetCP1256.html,
fix-for-TIKA2592-contributed-by-Andreas-Meier.patch
>
>
> HTML files are detected as utf-16 when meta content is set to "unicode".
> {code:XML}
> <meta http-equiv="Content-Type" content="text/html; charset="unicode">
>  {code}
>  
> Shouldn't the default be utf-8?
> The attached sample file is shown correctly in:
> Chromium Version 55.0.2883.75
> Firefox 50.1.0
> IE 11
> I am aware that there is no charset "unicode" (available character encodings: [http://www.iana.org/assignments/character-sets/character-sets.xhtml|http://www.iana.org/assignments/character-sets/character-sets.xhtml])
> Unfortunately there are many wrong encodings used out there.
> All unknown encodings should be validated or at least be set to default utf-8.
> Regards 
> Andreas



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message