nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Nioche (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (NUTCH-1634) readdb -stats show the result twice
Date Fri, 30 May 2014 14:51:04 GMT

     [ https://issues.apache.org/jira/browse/NUTCH-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Nioche resolved NUTCH-1634.
----------------------------------

    Resolution: Fixed

Committed revision 1598619.

Thanks Kaveh!

> readdb -stats show the result twice
> -----------------------------------
>
>                 Key: NUTCH-1634
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1634
>             Project: Nutch
>          Issue Type: Bug
>          Components: crawldb
>    Affects Versions: 2.2.1
>            Reporter: kaveh minooie
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: NUTCH-1634-2.x.patch
>
>
> right now this is the ouput:
> WebTable statistics start
> Statistics for WebTable: 
> status 2 (status_fetched):	1115
> min score:	0.0
> retry 2:	2
> retry 0:	11369
> jobs:	{db_stats-job_local1037462743_0001={jobID=job_local1037462743_0001, jobName=db_stats,
counters={File Input Format Counters ={BYTES_READ=0}, Map-Reduce Framework={MAP_OUTPUT_MATERIALIZED_BYTES=209,
MAP_INPUT_RECORDS=11376, REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24, MAP_OUTPUT_BYTES=602928,
COMMITTED_HEAP_BYTES=1181220864, CPU_MILLISECONDS=0, SPLIT_RAW_BYTES=757, COMBINE_INPUT_RECORDS=45504,
REDUCE_INPUT_RECORDS=12, REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0,
REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=45504}, FileSystemCounters={FILE_BYTES_READ=1819,
FILE_BYTES_WRITTEN=166199}, File Output Format Counters ={BYTES_WRITTEN=373}}}}
> retry 1:	5
> status 5 (status_redir_perm):	69
> max score:	1.0
> TOTAL urls:	11376
> status 3 (status_gone):	3
> status 4 (status_redir_temp):	3
> status 1 (status_unfetched):	10186
> avg score:	0.00342827
> WebTable statistics: done
> status 2 (status_fetched):	1115
> min score:	0.0
> retry 2:	2
> retry 0:	11369
> jobs:	{db_stats-job_local1037462743_0001={jobID=job_local1037462743_0001, jobName=db_stats,
counters={File Input Format Counters ={BYTES_READ=0}, Map-Reduce Framework={MAP_OUTPUT_MATERIALIZED_BYTES=209,
MAP_INPUT_RECORDS=11376, REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24, MAP_OUTPUT_BYTES=602928,
COMMITTED_HEAP_BYTES=1181220864, CPU_MILLISECONDS=0, SPLIT_RAW_BYTES=757, COMBINE_INPUT_RECORDS=45504,
REDUCE_INPUT_RECORDS=12, REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0,
REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=45504}, FileSystemCounters={FILE_BYTES_READ=1819,
FILE_BYTES_WRITTEN=166199}, File Output Format Counters ={BYTES_WRITTEN=373}}}}
> retry 1:	5
> status 5 (status_redir_perm):	69
> max score:	1.0
> TOTAL urls:	11376
> status 3 (status_gone):	3
> status 4 (status_redir_temp):	3
> status 1 (status_unfetched):	10186
> avg score:	0.00342827
> imho, it should be this:
> WebTable statistics start
> Statistics for WebTable: 
> status 2 (status_fetched):	1115
> min score:	0.0
> retry 2:	2
> retry 0:	11369
> jobs:	{db_stats-job_local801282144_0001={jobID=job_local801282144_0001, jobName=db_stats,
counters={File Input Format Counters ={BYTES_READ=0}, Map-Reduce Framework={MAP_OUTPUT_MATERIALIZED_BYTES=209,
MAP_INPUT_RECORDS=11376, REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24, MAP_OUTPUT_BYTES=602928,
COMMITTED_HEAP_BYTES=1122631680, CPU_MILLISECONDS=0, SPLIT_RAW_BYTES=757, COMBINE_INPUT_RECORDS=45504,
REDUCE_INPUT_RECORDS=12, REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0,
REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=45504}, FileSystemCounters={FILE_BYTES_READ=1819,
FILE_BYTES_WRITTEN=166191}, File Output Format Counters ={BYTES_WRITTEN=373}}}}
> retry 1:	5
> status 5 (status_redir_perm):	69
> max score:	1.0
> TOTAL urls:	11376
> status 3 (status_gone):	3
> status 4 (status_redir_temp):	3
> status 1 (status_unfetched):	10186
> avg score:	0.00342827
> WebTable statistics: done



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message