nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (NUTCH-1251) Deletion of duplicates fails with org.apache.solr.client.solrj.SolrServerException
Date Tue, 03 Apr 2012 11:32:25 GMT

     [ https://issues.apache.org/jira/browse/NUTCH-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Markus Jelsma updated NUTCH-1251:
---------------------------------

    Fix Version/s:     (was: 1.5)
                   1.6

20120304-push-1.6
                
> Deletion of duplicates fails with org.apache.solr.client.solrj.SolrServerException
> ----------------------------------------------------------------------------------
>
>                 Key: NUTCH-1251
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1251
>             Project: Nutch
>          Issue Type: Bug
>          Components: indexer
>    Affects Versions: 1.4
>         Environment: Any crawl where the number of URLs in Solr exceeds 1024 (the default
max number of clusters in Lucene boolean query).  
>            Reporter: Arkadi Kosmynin
>            Priority: Critical
>             Fix For: 1.6
>
>
> Deletion of duplicates fails. This happens because the "get all" query used to get Solr
index size is "id:[* TO *]", which is a range query. Lucene is trying to expand it to a Boolean
query and gets as many clauses as there are ids in the index. This is too many in a real situation
and it throws an exception. 
> To correct this problem, change the "get all" query (SOLR_GET_ALL_QUERY) to "\*:\*",
which is the standard Solr "get all" query.
> Indexing log extract:
> java.io.IOException: org.apache.solr.client.solrj.SolrServerException: Error executing
query
> 	at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:236)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> Caused by: org.apache.solr.client.solrj.SolrServerException: Error executing query
> 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
> 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
> 	at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:234)
> 	... 3 more
> Caused by: org.apache.solr.common.SolrException: Internal Server Error
> Internal Server Error
> request: http://localhost:8081/arch/select?q=id:[* TO *]&fl=id,boost,tstamp,digest&start=0&rows=82938&wt=javabin&version=2
> 	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
> 	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
> 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
> 	... 5 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message