nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2042) parse-html increase chunk size used to detect charset
Date Tue, 08 Dec 2015 23:01:11 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047640#comment-15047640
] 

Hudson commented on NUTCH-2042:
-------------------------------

SUCCESS: Integrated in Nutch-trunk #3326 (See [https://builds.apache.org/job/Nutch-trunk/3326/])
NUTCH-2042 parse-html increase chunk size used to detect charset (snagel: [http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1718718])
* trunk/CHANGES.txt
* trunk/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java


> parse-html increase chunk size used to detect charset
> -----------------------------------------------------
>
>                 Key: NUTCH-2042
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2042
>             Project: Nutch
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 2.3, 1.10
>            Reporter: Sebastian Nagel
>            Assignee: Sebastian Nagel
>            Priority: Minor
>             Fix For: 2.3.1, 1.12
>
>         Attachments: NUTCH-2042-2x-v1.patch, NUTCH-2042-trunk-v1.patch, NUTCH-2042-trunk-v2.patch
>
>
> The chunk used to detect the encoding of a document is set to 2000 bytes. Although it
is definitely best practice to "define" the character set on top, 2000 bytes are sometimes
not enough: 20 longer <link> elements pointing to javascript and css libs may "hide"
the <meta> element containing content type and encoding. Same problem has been observed
in TIKA-357 and solved by increasing the buffer size to 8 kB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message