nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dogacan Güney (JIRA) <>
Subject [jira] Commented: (NUTCH-443) allow parsers to return multiple Parse object, this will speed up the rss parser
Date Thu, 08 Feb 2007 10:17:07 GMT


Dogacan Güney commented on NUTCH-443:

Ok, this is  the second attempt(sorry that I am sending patches in a frenzy, I will slow down

In the first patch, I just put Map<String, Parse> to FetcherOutput but that doesn't
since the keys are not necessarily ordered.

I mean;
Assume, we have two <key, Map<String, Parse>> pairs:
<"", <"", some_parse>> and <"", <"", some_other_parse>>
With the first patch, we would first get (and thus write then get (and
try to write
but this would fail since "" < "".

I completely removed FetcherOutput class. What it does can be done with wrapping the objects

in ObjectWritable. I know this is heavier but I couldn't think of another way around the issue
of ordering
of keys.

I tested this a bit with a small set of urls. Both parsing seperately and parsing during fetching
seems to work.

> allow parsers to return multiple Parse object, this will speed up the rss parser
> --------------------------------------------------------------------------------
>                 Key: NUTCH-443
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: Renaud Richardet
>            Priority: Minor
>             Fix For: 0.9.0
>         Attachments: parse-map-core-untested.patch
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser can
return multiple parse objects, that will all be indexed separately. Advantage: no need to
fetch all feed-items separately.
> see the discussion at

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message