nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2328) GeneratorJob does not generate anything on second run
Date Wed, 19 Oct 2016 13:23:58 GMT


Sebastian Nagel commented on NUTCH-2328:

Hi [~arthur-evozon],

> Btw., I'm even not sure whether counters in the task context are updated with global
job values.

The answer is: counters in task context *do not show aggregated values* over all tasks of
a job.
That's more or less clear from these definitions: "Counter is a facility for MapReduce applications
to report its statistics." [[1|]],
more details in Tom White's "Hadoop, the definitive guide". Counter values are propagated
and aggregated in this direction: task > task tracker > job tracker > job client.
The aggregated values are not sent back. There are hacks to access the aggregated values from
the tasks, cf. [[2|]].
But the central problem persists: the value of a job counter depends mostly on how many tasks
are in progress or already done. If the programming logic relies on such a value it becomes
largely indeterministic.

> GeneratorJob does not generate anything on second run
> -----------------------------------------------------
>                 Key: NUTCH-2328
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: generator
>    Affects Versions: 2.2, 2.3, 2.2.1, 2.3.1
>         Environment: Ubuntu 16.04 / Hadoop 2.7.1
>            Reporter: Arthur B
>              Labels: fails, generator, subsequent
>             Fix For: 2.4
>         Attachments: generator-issue-static-count.patch
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> Given a topN parameter (ie 10) the GeneratorJob will fail to generate anything new on
the subsequent runs within the same process space.
> To reproduce the issue submit the GeneratorJob twice one after another to the M/R framework.
Second time will say it generated 0 URLs.
> This issue is due to the usage of the static count field (org.apache.nutch.crawl.GeneratorReducer#count)
to determine if the topN value has been reached.

This message was sent by Atlassian JIRA

View raw message