tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ken Krugler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (TIKA-357) Increase buffer size for meta tag sniffing
Date Mon, 18 Jan 2010 05:08:54 GMT

    [ https://issues.apache.org/jira/browse/TIKA-357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801679#action_12801679
] 

Ken Krugler commented on TIKA-357:
----------------------------------

Hi Chris,

This patch fixes an issue with detecting the charset for an HTML document, not detecting whether
a document is HTML vs. plain text.

It changes the amount of text to "sniff" when trying to find a <meta> tag that contains
charset info for an http-equiv value.

You've run into a separate issue that I've seen, where the AutoDetection code fails on pages
that have big Javascript blocks at the beginning.


> Increase buffer size for meta tag sniffing
> ------------------------------------------
>
>                 Key: TIKA-357
>                 URL: https://issues.apache.org/jira/browse/TIKA-357
>             Project: Tika
>          Issue Type: Improvement
>    Affects Versions: 0.5
>            Reporter: Ken Krugler
>            Assignee: Chris A. Mattmann
>            Priority: Minor
>             Fix For: 0.6
>
>         Attachments: makler.html, TIKA-357.patch
>
>
> Some web pages (such as makler.su, see attached) have lots of script data before the
body of the HTML.
> When this happens, the sniffing code fails to find the charset info in the meta tag,
because it currently only sniffs the first 4K.
> Bumping it to 8K would cover all of the cases that I (Ken) have seen during a test crawl.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message