nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jianyun He (JIRA)" <j...@apache.org>
Subject [jira] [Created] (NUTCH-1416) Can not update the index
Date Sun, 01 Jul 2012 08:10:49 GMT
Jianyun He created NUTCH-1416:
---------------------------------

             Summary: Can not update the index
                 Key: NUTCH-1416
                 URL: https://issues.apache.org/jira/browse/NUTCH-1416
             Project: Nutch
          Issue Type: Bug
          Components: indexer
            Reporter: Jianyun He


When we update the index,can not guarantee that the contents which be indexed is the latest.In
the class IndexerMapReduce and method reduce(), it has the following code:
public void reduce(Text key, Iterator<NutchWritable> values,
                     OutputCollector<Text, NutchDocument> output, Reporter reporter)
throws IOException {
   ……
   } else if (value instanceof ParseData) {  
        parseData = (ParseData)value;
   } else if (value instanceof ParseText) { 
        parseText = (ParseText)value;
   }
   ……
}
For example,30 days ago,I fetched the web page A,now I fetch it again. Then the key A will
correspond to two ParseData objects(located in different segments).But in this code,it does
not compare the fetch time and simply overwrites the previous value.So the final value maybe
the old one.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message