nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2456) Allow to index pages/URLs not contained in CrawlDb
Date Wed, 08 Nov 2017 21:35:00 GMT


Markus Jelsma commented on NUTCH-2456:

What will this patch achieve then? Just the case of ignoring dbDatum i presume? If have a
hard time reading githubs output here, my problem.

How about index.*.md? What will happen if someone has both and
enabled? Will it end up with duplicates, fields not configured to be multivalued? If so, that
would need a new issue for that.

The problem of duplicates and orphans is obvious. But that can be remedied by never having
those eligible for fetch via a custom (or built-in) FetchSchedule.

To be clear, i am really for having this feature as it will reduce cycle time by a lot for
large CrawlDBs, especially if you start indexing in parallel when updatedb is due.

> Allow to index pages/URLs not contained in CrawlDb
> --------------------------------------------------
>                 Key: NUTCH-2456
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: indexer
>    Affects Versions: 1.13
>            Reporter: Yossi Tamari
>            Priority: Critical
> If http.redirect.max is set to a positive value, the Fetcher will follow redirects, creating
a new CrawlDatum.
> If the redirected URL is fetched and parsed, during indexing for it we have a special
case: dbDatum is null. This means that in []
the document is not indexed, as it is assumed it only has inlinks (actually it has everything
but dbDatum).
> I'm not sure what the correct fix is here. It seems to me the condition should use AND
instead of OR anyway, but I may not understand the original intent. It is clear that it is
too strict as is.
> However, the code following that line assumes all 4 objects are not null, so a patch
would need to change more than just the condition.

This message was sent by Atlassian JIRA

View raw message