nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Sullivan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (NUTCH-1475) Nutch 2.1 Index-More Plugin -- A better fall back value for date field
Date Sun, 07 Oct 2012 01:51:02 GMT
James Sullivan created NUTCH-1475:
-------------------------------------

             Summary: Nutch 2.1 Index-More Plugin -- A better fall back value for date field
                 Key: NUTCH-1475
                 URL: https://issues.apache.org/jira/browse/NUTCH-1475
             Project: Nutch
          Issue Type: Bug
    Affects Versions: nutchgora, 2.1
         Environment: All
            Reporter: James Sullivan
            Priority: Minor
         Attachments: index-more-2x.patch

Among other fields, the more plugin for Nutch 2.x provides a "last modified" and "date" field
for the Solr index. The "last modified" field is the last modified date from the http headers
if available, if not available it is left empty. Currently, the "date" field is the same as
the "last modified" field unless that field is empty in which case getFetchTime is used as
a fall back. I think getFetchTime is not a good fall back as it is the next fetch time and
often a month or more in the future which doesn't make sense for the date field. Users do
not expect webpages/documents with future dates. A more sensible fallback would be current
date at the time it is indexed. 

This is possible by simply changing line 97 of https://svn.apache.org/repos/asf/nutch/branches/2.x/src/plugin/index-more/src/java/org/apache/nutch/indexer/more/MoreIndexingFilter.java
from


time = page.getFetchTime(); // use fetch time

to

time = new Date().getTime();


Users interested in the getFetchTime value can still get it from the "tstamp" field.




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message