nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dogacan Güney (JIRA) <>
Subject [jira] Updated: (NUTCH-443) allow parsers to return multiple Parse object, this will speed up the rss parser
Date Fri, 09 Feb 2007 22:41:06 GMT


Dogacan Güney updated NUTCH-443:

    Attachment: NUTCH-443-draft-v3.patch

new patch, contains a possible fix for CrawlDbReducer problem.

This version finally works! (well, not really, but I can definitely say that it almost kind
of works..sometimes:)

I have two main issues with this patch:

1) If fetcher is in parsing mode, and parse returns a SUCCESS_REDIRECT,
fetcher handles this redirect. After this change, fetcher checks if the first element of parseMap.values()
(whatever that may be) has a SUCCESS_REDIRECT. It is possible that a multi-entry parseMap
has an parse element with a SUCCESS_REDIRECT that is not the first element. (perhaps we can
first check if parseMap.get(originalUrl) returns a parse, if not use first element of parseMap.values()?

2) To be able to pass fetch time to not-actually-fetched-but-generated-in-parse urls, I first
put the original fetch time to content and then pass the value in content to all elements
in parseMap.values(). I guess this approach is not very optimal since it passes fetch time
around a lot.

> allow parsers to return multiple Parse object, this will speed up the rss parser
> --------------------------------------------------------------------------------
>                 Key: NUTCH-443
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: Renaud Richardet
>            Priority: Minor
>             Fix For: 0.9.0
>         Attachments: NUTCH-443-draft-v1.patch, NUTCH-443-draft-v2.patch, NUTCH-443-draft-v3.patch,
parse-map-core-draft-v1.patch, parse-map-core-untested.patch, parsers.diff
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser can
return multiple parse objects, that will all be indexed separately. Advantage: no need to
fetch all feed-items separately.
> see the discussion at

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message