nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (NUTCH-1074) topN is ignored with maxNumSegments
Date Wed, 07 Sep 2011 09:57:09 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098820#comment-13098820
] 

Markus Jelsma edited comment on NUTCH-1074 at 9/7/11 9:56 AM:
--------------------------------------------------------------

Finally got some numbers to share from a running test:

maxNumSegments = 3
topN = 250.000
Selector reduce output records = 750.000 
happen
The above looks fine. The generator selects exactly numSegments * topN records to be consumed
by the following numSegments partitioners. Here's the number of output reducer records of
the following three partitioned segments in order:

1: 471.428
2: 171.562
3: 107.010

The strange thing is that the number of reduce output records exactly matches the total number
of map input records. This is not what i had expected. The generator partitions by host and
has a limit on the number of hosts per queue so i would expect each segment to contain slightly
less records than topN but certainly not more than topN.

What should we exactly expect?



      was (Author: markus17):
    Finally got some numbers to share from a running test:

maxNumSegments = 3
topN = 250.000
Selector reduce output records = 750.000 
happen
The above looks fine. The generator selects exactly numSegments * topN records to be consumed
by the following numSegments partitioners. Here's the number of output reducer records of
the following three partitioned segments in order:

1: 471.428
2: 171.562
3: 107.010

The strange thing is that the number of reduce output records exactly matches the total number
of map input records. This is not what i had expected. The generator partitions by host and
has a limit on the number of hosts per queue so i would expect each segment to contain slightly
less records than topN.

What should we exactly expect?


  
> topN is ignored with maxNumSegments
> -----------------------------------
>
>                 Key: NUTCH-1074
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1074
>             Project: Nutch
>          Issue Type: Bug
>          Components: generator
>    Affects Versions: 1.3
>            Reporter: Markus Jelsma
>             Fix For: 1.4
>
>
> When generating segments with topN and maxNumSegments, topN is not respected. It looks
like the first generated segment contains topN * maxNumSegments of URLs's, at least the number
of map input records roughly matches.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message