nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AJ Chen <cano...@gmail.com>
Subject Re: fetch performance
Date Sat, 10 Sep 2005 22:22:15 GMT
Nutch 0.7 default plugin-includes property does not include 
protocol-httpclient. After it's added, crawling does recognize https 
urls.  Thanks.  However, there are still two kinds of error related to 
https.

(1) NoRouteToHostException.  It occurs very often, for example,

050910 150336 fetching https://www.picoscript.com/products.aspx
050910 150336 fetch of https://www.picoscript.com/products.aspx failed 
with: java.lang.Exception: java.net.NoRouteToHost
Exception: No route to host: connect

(2) does not recognize https url redirected from http url. It occurs 
very often. for example,

050910 150341 fetch of 
http://www.cellsciences.com/content/c2-contact.asp failed with: 
java.lang.Exception: org.apache.n
utch.protocol.http.HttpException: Not an HTTP 
url:https://www.cellsciences.com/content/c2-contact.asp

Any idea what happens?

-AJ

Andrzej Bialecki wrote:

> AJ Chen wrote:
>
>> Andrzej, Thanks.
>> A related question: Some of the sites I crawl use https: or redirect 
>> to https:.  Nutch default setting does not recognize https: as valid 
>> url. Is there a way to crawl url starting with "https:"?
>
>
> Which version of Nutch? 0.7 recognizes and supports https urls, 
> through the protocol-httpclient plugin.
>


Mime
View raw message