nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lewis John McGibbney (Commented) (JIRA)" <>
Subject [jira] [Commented] (NUTCH-990) protocol-httpclient fails with short pages
Date Wed, 09 Nov 2011 18:47:51 GMT


Lewis John McGibbney commented on NUTCH-990:

Hi this issue has been marked as fixed but not closed off. Should it be marked as won't fix
as its a duplication then closed off as per Juliens comments above? I think we need to remove/resolve
some of the issues muddying this umbrella topic of problems with protocol-httpclient
> protocol-httpclient fails with short pages
> ------------------------------------------
>                 Key: NUTCH-990
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>            Reporter: Gabriele Kahlout
>            Priority: Minor
>             Fix For: 1.4
>         Attachments: hadoop.log
> Using protocol-http with a few words html pages works fine. But with protocol-httpclient
the same pages disappear from the index, although they are still fetched.
> Those small files are useful for quick testing. 
> Steps to reproduce:
> $ svn co nutch-1.3
> Checked out revision 1097214.
> $ cd nutch-1.3
> $ xmlstarlet edit -L -u "/configuration/property[name='']"/value -v 'test'
> $ ant
> Download to runtime/local the following script and seeds list file. They assume a $HADOOP_HOME
environment variable. It's a 1.3 adaptation of [1].
> $ cd runtime/local
> This will empty your Solr index (-f) and crawl:
> $ ./whole-web-crawling-incremental -f .
> Now Check Solr index searching for artificial and you will find the page pointed to in
> Now change plugin-includes in conf/nutch-default to use protocol-httpclient instead of
protocol-http and re-run the script. No more results in solr. Try again with http and the
results return.
> [1]

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message