lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Heisey" <s...@elyograg.org>
Subject Re: Streaming Updates Using HttpSolrServer.add(Iterator) In Solr 4.3
Date Tue, 30 Jul 2013 00:10:46 GMT
> I am indexing more than 300 million records, it takes less than 7 hours to
> index all the records..
>
> Send the documents in batches and also use CUSS
> (ConcurrentUpdateSolrServer)
> for multi threading support.
>
> Ex:
>
>  ConcurrentUpdateSolrServer server= new
> ConcurrentUpdateSolrServer(solrServer, queueSize,
> 					threadCount);
> 		List<SolrInputDocument> solrDocList = new
> ArrayList<SolrInputDocument>();
>      While (loop) {
> 	solrDocList.add(doc); --> Add the documents to array
>        if(count >=100){
>            server.add(solrDocList); --> Add documents to SOLR in batches
>        }
>      count++;
>     }
> server.commit(); --> Commit after adding all the documents

Using CUSS is only acceptable if you don't care about error handling. If
you shut down the Solr server, your program will only see an error on the
commit. It will think the update worked perfectly, even though the server
is down.




Mime
View raw message