nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniele Menozzi <>
Subject Re: what contibute to fetch slowing down
Date Wed, 28 Sep 2005 20:41:30 GMT
On  10:27:55 28/Sep , AJ Chen wrote:
> I started the crawler with about 2000 sites.  The fetcher could achieve 
> 7 pages/sec initially, but the performance gradually dropped to about 2 
> pages/sec, sometimes even 0.5 pages/sec.  The fetch list had 300k pages 
> and I used 500 threads. What are the main causes of this slowing down? 

I have the same problem; I've tried with different number of fetchers
(10,20,50,100,..), but the download rate always decrease sistematically,
page after page.
The machine is a p4 1.7, 768 MB ram, running debian on 2.6.12 kernel.
The bandwidth isn't a problem (10Mbit in and 10Mbit out), but I cannot
obtain a stable, and high, page/s rate.
I've also tried to change machine and kernel, but the problem still
Can you please give us some advice?
Thank you for your help,

		      Free Software Enthusiast
		 Debian Powered Linux User #332564

View raw message