tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksandr Dubinsky (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (TIKA-1406) Problems in TXT encoding detection
Date Sun, 31 Aug 2014 12:25:20 GMT

     [ https://issues.apache.org/jira/browse/TIKA-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aleksandr Dubinsky updated TIKA-1406:
-------------------------------------
    Attachment: 0001-fix-TXT-encoding-detection.patch

> Problems in TXT encoding detection
> ----------------------------------
>
>                 Key: TIKA-1406
>                 URL: https://issues.apache.org/jira/browse/TIKA-1406
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.6, 1.7
>            Reporter: Aleksandr Dubinsky
>         Attachments: 0001-fix-TXT-encoding-detection.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The detection of TXT file encoding often makes mistakes. Two things can improve the detection
in many cases:
> - Increase the lookahead from 16k bytes to a larger number, such as 128k or larger.
> - Improve on the brain-dead heuristic that if a file doesn't have a \r then it must be
ISO 8859-1(5) instead of Windows-1252. (For one, it mis-detects files that don't have any
newlines.)
> A remaining problem that doesn't have an immediate solution, is the frequent mis-detection
of Windows-1252 as Shift-JIS. A flag to forbit Shift-JIS is desirable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message