nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-1052) Multiple deletes of the same URL using SolrClean
Date Tue, 20 Sep 2011 12:54:09 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108641#comment-13108641
] 

Markus Jelsma commented on NUTCH-1052:
--------------------------------------

Thanks for your comments! Just to make sure i understand you correctly. You agree we should
use a container object that holds the NutchDocument and the action (ADD || DELETE) and pass
it to RecordWriter.write() instead of abusing NULL encoded as delete action as i do it now?

Then i'd need to add a class somewehere such as:

{code}
class NutchIndexAction {
  public static enum Action ADD,DELETE;

  public NutchDocument doc;
  public Action action;
}
{code}

And pass a NutchIndexAction instance to the record writer from IndexerMapReduce? If so, what
would be the appropriate location for such a class?

> Multiple deletes of the same URL using SolrClean
> ------------------------------------------------
>
>                 Key: NUTCH-1052
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1052
>             Project: Nutch
>          Issue Type: Improvement
>          Components: indexer
>    Affects Versions: 1.3, 1.4
>            Reporter: Tim Pease
>            Assignee: Julien Nioche
>             Fix For: 1.4, 2.0
>
>         Attachments: NUTCH-1052-1.4-1.patch, NUTCH-1052-1.4-2.patch, NUTCH-1052-1.4-3.patch
>
>
> The SolrClean class does not keep track of purged URLs, it only checks the URL status
for "db_gone". When run multiple times the same list of URLs will be deleted from Solr. For
small, stable crawl databases this is not a problem. For larger crawls this could be an issue.
SolrClean will become an expensive operation.
> One solution is to add a "purged" flag in the CrawlDatum metadata. SolrClean would then
check this flag in addition to the "db_gone" status before adding the URL to the delete list.
> Another solution is to add a new state to the status field "db_gone_and_purged".
> Either way, the crawl DB will need to be updated after the Solr delete has successfully
occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message