nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doğacan Güney (JIRA) <>
Subject [jira] Commented: (NUTCH-719) fetchQueues.totalSize incorrect in Fetcher2
Date Fri, 10 Jul 2009 21:35:14 GMT


Doğacan Güney commented on NUTCH-719:

Thanks for looking into this bug.

I wonder if this is the cause of the performance problem so many people are facing with Fetcher
in nutch-1.0. Can it be that QueueFeeder stops feeding new URLs into FetchQueues because of
this bug?

> fetchQueues.totalSize incorrect in Fetcher2
> -------------------------------------------
>                 Key: NUTCH-719
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Julien Nioche
> I had a look at the logs generated by Fetcher2 and found cases where there were no active
fetchQueues but fetchQueues.totalSize was != 0
> fetcher.Fetcher2 - -activeThreads=200, spinWaiting=200, fetchQueues.totalSize=1, fetchQueues=0
> since the code relies on fetchQueues.totalSize to determine whether the work is finished
or not the task is blocked until the abortion mechanism kicks in
> 2009-03-12 09:27:38,977 WARN  fetcher.Fetcher2 - Aborting with 200 hung threads.
> could that be a synchronisation issue? any ideas?

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message