nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uros Gruber (JIRA)" <>
Subject [jira] Commented: (NUTCH-361) generator create fetchlist randomly
Date Wed, 06 Sep 2006 14:27:27 GMT
    [ ] 
Uros Gruber commented on NUTCH-361:

Here is my latest findings. While debuging TestGenerator I found  that in org.apache.nutch.crawl.PartitionUrlByHost.getPartition
numReduceTasks became 0 which cause java.lang.ArithmeticException: / by zero. My patch does
not solve the problem. 

One thing I can't understand is. First thread goes through all 100 urls and in log I see 
map 100%  reduce 0% after that new thread is started from 0% but this thread is run with numReduceTasks
to zero.

All this is tested with released hadoop 0.5.0 also bundled with SVN nutch.

I don't know If I'm the only one seeing this bug, because no one reply.

> generator create fetchlist randomly
> -----------------------------------
>                 Key: NUTCH-361
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.9.0
>         Environment: Java 1.5, FreeBSD 6.1
>            Reporter: Uros Gruber
>            Priority: Critical
>         Attachments: partition.diff
> I noticed problems during generating fetchlist. I already post some info at the users
list. Today I check release 0.8 and I'm certain that problem is only in version later than
this. I've do testnig only on 0.8 and svn from today.
> The problem is that generator generate fetchlist from crawldb but everytime i run there
is different number of urls in fetchlist.
> For example I put 6 test urls we have for testing and only 5 of 20 test there were all
urls listed in fetchlist, sometimes onyl one. Config was always the same also when testing
at version 0.8.
> I try to debug what might go wrong but I only end up that in /tmp there were all urls
but somehow missed in crawl_generate
> I also se some of 
> 2006-09-02 20:14:20,147 DEBUG conf.Configuration - config(config)
>         at org.apache.hadoop.conf.Configuration.<init>(
>         at org.apache.hadoop.mapred.JobConf.<init>(
>         at org.apache.hadoop.mapred.JobConf.<init>(
>         at org.apache.nutch.util.NutchJob.<init>(
>         at org.apache.nutch.crawl.Generator.generate(
>         at
>         at org.apache.nutch.util.ToolBase.doMain(
>         at org.apache.nutch.crawl.Generator.main(
> if I enable DEBUG loging but I doubt that this has anything to do with this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:


View raw message