nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (JIRA)" <>
Subject [jira] [Commented] (NUTCH-1067) Configure minimum throughput for fetcher
Date Wed, 14 Sep 2011 12:05:09 GMT


Markus Jelsma commented on NUTCH-1067:

* Crawl and Benchmark both read the value and pass it to the fetcher. It's safe there to remove
the argument.

* There's also a problem with TestFetcher.testFetch(). This it seems, relies on the parse
to work as it passes TRUE to the fetcher but doesn't set the directive. I'll override the
configuration directive to TRUE there.

* TestFetcher.testAgentNameCheck() for some reason sets the conf directive to FALSE but passes
TRUE as argument.

All source code and tests now compile again. The fetcher tests also pass without errors. 
I'll attach a patch now.

> Configure minimum throughput for fetcher
> ----------------------------------------
>                 Key: NUTCH-1067
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>            Priority: Minor
>             Fix For: 1.4
>         Attachments: NUTCH-1067-1.4-1.patch, NUTCH-1067-1.4-2.patch, NUTCH-1067-1.4-3.patch,
> Large fetches can contain a lot of url's for the same domain. These can be very slow
to crawl due to politeness from robots.txt, e.g. 10s per url. If all other url's have been
fetched, these queue's can stall the entire fetcher, 60 url's can then take 10 minutes or
even more. This can usually be dealt with using the time bomb but the time bomb value is hard
to determine.
> This patch adds a fetcher.throughput.threshold setting meaning the minimum number of
pages per second before the fetcher gives up. It doesn't use the global number of pages /
running time but records the actual pages processed in the previous second. This value is
compared with the configured threshold.
> Besides the check the fetcher's status is also updated with the actual number of pages
per second and bytes per second.

This message is automatically generated by JIRA.
For more information on JIRA, see:


View raw message