nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nutch.newbie (JIRA)" <>
Subject [jira] Commented: (NUTCH-443) allow parsers to return multiple Parse object, this will speed up the rss parser
Date Sat, 10 Feb 2007 16:04:07 GMT


nutch.newbie commented on NUTCH-443:


It works rather ok, But hen I changed  the parse-plugins.xml a bit  just to see if I was explicitly
using the parse-rss ..

        <mimeType name="text/xml">
<!--            <plugin id="parse-html" /> -->
                <plugin id="parse-rss" />

I got stuck in dedup phase .. I tried couple of time with content type magic being on/off
 but same error.. stack trace..

2007-02-10 16:48:35,873 DEBUG mapred.MapTask - Started thread: Sort progress reporter for
task map_e7j0jb
2007-02-10 16:48:35,877 WARN  mapred.LocalJobRunner - job_kyv1oj
java.lang.ArrayIndexOutOfBoundsException: -1
        at org.apache.lucene.index.MultiReader.isDeleted(
        at org.apache.nutch.indexer.DeleteDuplicates$InputFormat$
        at org.apache.hadoop.mapred.MapTask$
        at org.apache.hadoop.mapred.LocalJobRunner$

I will be doing some more testing tonight with various config changes.. But in general without
any changes it works. I however haven't tested searching yet.

Please advise me if I am doing something wrong or if you want me to test something specifically.

> allow parsers to return multiple Parse object, this will speed up the rss parser
> --------------------------------------------------------------------------------
>                 Key: NUTCH-443
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: Renaud Richardet
>         Assigned To: Chris A. Mattmann
>            Priority: Minor
>             Fix For: 0.9.0
>         Attachments: NUTCH-443-draft-v1.patch, NUTCH-443-draft-v2.patch, NUTCH-443-draft-v3.patch,
NUTCH-443-draft-v4.patch, parse-map-core-draft-v1.patch, parse-map-core-untested.patch, parsers.diff
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser can
return multiple parse objects, that will all be indexed separately. Advantage: no need to
fetch all feed-items separately.
> see the discussion at

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message