nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doğacan Güney (JIRA) <j...@apache.org>
Subject [jira] Commented: (NUTCH-719) fetchQueues.totalSize incorrect in Fetcher2
Date Fri, 10 Jul 2009 21:35:14 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729833#action_12729833
] 

Doğacan Güney commented on NUTCH-719:
-------------------------------------

Thanks for looking into this bug.

I wonder if this is the cause of the performance problem so many people are facing with Fetcher
in nutch-1.0. Can it be that QueueFeeder stops feeding new URLs into FetchQueues because of
this bug?

> fetchQueues.totalSize incorrect in Fetcher2
> -------------------------------------------
>
>                 Key: NUTCH-719
>                 URL: https://issues.apache.org/jira/browse/NUTCH-719
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Julien Nioche
>
> I had a look at the logs generated by Fetcher2 and found cases where there were no active
fetchQueues but fetchQueues.totalSize was != 0
> fetcher.Fetcher2 - -activeThreads=200, spinWaiting=200, fetchQueues.totalSize=1, fetchQueues=0
> since the code relies on fetchQueues.totalSize to determine whether the work is finished
or not the task is blocked until the abortion mechanism kicks in
> 2009-03-12 09:27:38,977 WARN  fetcher.Fetcher2 - Aborting with 200 hung threads.
> could that be a synchronisation issue? any ideas?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message