nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MilleBii <mille...@gmail.com>
Subject Re: [jira] Commented: (NUTCH-776) Configurable queue depth
Date Thu, 07 Jan 2010 17:55:12 GMT
Actually I created a key to set it adequately... The best results came
with a depth of 1 and a big number of threads (I use 1800) ?!?
That is because I have numerous sites (like blogs) that have different
domain names and single IP... This a result of topical focused
crawling.


Since it was not enough speed, I decided to allow fetching on same
site every second... Works fine although not according to
netetiquette.

We should still create this key because it is very handy when trying
to optimize.

In terms of ressources we should be explicit it consummes #Threads x
#depth ... On my 4GB it was saturating around 4000 total Q size.

2010/1/7, Julien Nioche (JIRA) <jira@apache.org>:
>
>     [
> https://issues.apache.org/jira/browse/NUTCH-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797653#action_12797653
> ]
>
> Julien Nioche commented on NUTCH-776:
> -------------------------------------
>
> Did you notice any improvement in the fetch rate after I suggested on the
> mailing list to use a value larger than 50? Does the memory consumption
> remain reasonable?
>
>> Configurable queue depth
>> ------------------------
>>
>>                 Key: NUTCH-776
>>                 URL: https://issues.apache.org/jira/browse/NUTCH-776
>>             Project: Nutch
>>          Issue Type: Improvement
>>          Components: fetcher
>>    Affects Versions: 1.1
>>            Reporter: MilleBii
>>            Priority: Minor
>>             Fix For: 1.1
>>
>>
>> I propose that we create a configurable item for the queuedepth in
>> Fetcher.java instead of the hard-coded value of 50.
>> key name : fetcher.queues.depth
>> Default value : remains 50 (of course)
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>


-- 
-MilleBii-

Mime
View raw message