nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lewis John McGibbney (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-990) protocol-httpclient fails with short pages
Date Wed, 09 Nov 2011 18:47:51 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147217#comment-13147217
] 

Lewis John McGibbney commented on NUTCH-990:
--------------------------------------------

Hi this issue has been marked as fixed but not closed off. Should it be marked as won't fix
as its a duplication then closed off as per Juliens comments above? I think we need to remove/resolve
some of the issues muddying this umbrella topic of problems with protocol-httpclient
                
> protocol-httpclient fails with short pages
> ------------------------------------------
>
>                 Key: NUTCH-990
>                 URL: https://issues.apache.org/jira/browse/NUTCH-990
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>            Reporter: Gabriele Kahlout
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: hadoop.log
>
>
> Using protocol-http with a few words html pages works fine. But with protocol-httpclient
the same pages disappear from the index, although they are still fetched.
> Those small files are useful for quick testing. 
> Steps to reproduce:
> $ svn co http://svn.apache.org/repos/asf/nutch/branches/branch-1.3 nutch-1.3
> Checked out revision 1097214.
> $ cd nutch-1.3
> $ xmlstarlet edit -L -u "/configuration/property[name='http.agent.name']"/value -v 'test'
conf/nutch-default.xml
> $ ant
> Download to runtime/local the following script and seeds list file. They assume a $HADOOP_HOME
environment variable. It's a 1.3 adaptation of [1].
> http://dp4j.sf.net/debug/whole-web-crawling-incremental
> http://dp4j.sf.net/debug/urls
> $ cd runtime/local
> This will empty your Solr index (-f) and crawl:
> $ ./whole-web-crawling-incremental -f .
> Now Check Solr index searching for artificial and you will find the page pointed to in
urls.
> Now change plugin-includes in conf/nutch-default to use protocol-httpclient instead of
protocol-http and re-run the script. No more results in solr. Try again with http and the
results return.
> [1] http://wiki.apache.org/nutch/Whole-Web%20Crawling%20incremental%20script

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message