nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Renaud Richardet (JIRA)" <j...@apache.org>
Subject [jira] Updated: (NUTCH-369) StringUtil.resolveEncodingAlias is unuseful.
Date Sat, 24 Feb 2007 08:25:05 GMT

     [ https://issues.apache.org/jira/browse/NUTCH-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Renaud Richardet updated NUTCH-369:
-----------------------------------

    Attachment: patch.diff

unified diff against head.

- fixes encoding, as described by King Kong
- removes non-valid features
- fixes logging

> StringUtil.resolveEncodingAlias  is unuseful.
> ---------------------------------------------
>
>                 Key: NUTCH-369
>                 URL: https://issues.apache.org/jira/browse/NUTCH-369
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.8
>         Environment: all
>            Reporter: King Kong
>         Attachments: patch.diff
>
>
> After we defined encoding alias map in StringUtil , but parse html use orginal encoding
also.
> I found it is reading charset from  meta in nekohtml which HtmlParser  used .
> we can set it's feature "http://cyberneko.org/html/features/scanner/ignore-specified-charset"
to true 
> that nekohtml will use encoding we set;
> concretely,
>   private DocumentFragment parseNeko(InputSource input) throws Exception {
>     DOMFragmentParser parser = new DOMFragmentParser();
>     // some plugins, e.g., creativecommons, need to examine html comments
>     try {
>    + parser.setFeature("http://cyberneko.org/html/features/scanner/ignore-specified-charset",true);
>       parser.setFeature("http://apache.org/xml/features/include-comments", 
>               true);
>       ....
> BTW, It must be add on front of try block,because the following sentence  (parser.setFeature("http://apache.org/xml/features/include-comments",

>               true);) will throw exception.
>      

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message