nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claudio Martella (JIRA)" <j...@apache.org>
Subject [jira] Commented: (NUTCH-963) Add support for deleting Solr documents with STATUS_DB_GONE in CrawlDB (404 urls)
Date Thu, 27 Jan 2011 16:06:45 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987617#action_12987617
] 

Claudio Martella commented on NUTCH-963:
----------------------------------------

@Markus: about the commit, i did also consider that, i just noticed that solrdeduplication
doesn't have it and put the code on the same line. Sounds reasonable to me too. For the name,
everything is fine for me.
Also the idea to delete redirects occurred to me, we could OR them in the Mapper.

@Julien: you mean to use the signature on the CrawlDB?

> Add support for deleting Solr documents with STATUS_DB_GONE in CrawlDB (404 urls)
> ---------------------------------------------------------------------------------
>
>                 Key: NUTCH-963
>                 URL: https://issues.apache.org/jira/browse/NUTCH-963
>             Project: Nutch
>          Issue Type: New Feature
>          Components: indexer
>    Affects Versions: 2.0
>            Reporter: Claudio Martella
>            Assignee: Markus Jelsma
>            Priority: Minor
>             Fix For: 1.3, 2.0
>
>         Attachments: Solr404Deleter.java
>
>
> When issuing recrawls it can happen that certain urls have expired (i.e. URLs that don't
exist anymore and return 404).
> This patch creates a new command in the indexer that scans the crawldb looking for these
urls and issues delete commands to SOLR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message