nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2463) Enable sampling CrawlDB
Date Tue, 28 Nov 2017 11:07:00 GMT


ASF GitHub Bot commented on NUTCH-2463:

YossiTamari commented on issue #243: NUTCH-2463 - Enable sampling CrawlDB

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

> Enable sampling CrawlDB
> -----------------------
>                 Key: NUTCH-2463
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: crawldb
>            Reporter: Yossi Tamari
>            Priority: Minor
>             Fix For: 1.14
> CrawlDB can grow to contain billions of records. When that happens *readdb -dump* is
pretty useless, and *readdb -topN* can run for ages (and does not provide a statistically
correct sample).
> We should add a parameter *-sample* to *readdb -dump* which is followed by a number between
0 and 1, and only that fraction of records from the CrawlDB will be processed.
> The sample should be statistically random, and all the other filters should be applied
on the sampled records.

This message was sent by Atlassian JIRA

View raw message