lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SolrLover <bbar...@gmail.com>
Subject Solr Cloud - How to balance Batch and Queue indexing?
Date Mon, 29 Jul 2013 23:26:09 GMT
I need some advice on the best way to implement Batch indexing with soft
commit / Push indexing (via queue) with soft commit when using SolrCloud.

*I am trying to figure out a way to:
*
1. Make the push indexing available almost real time (using soft commit)
without degrading the search / indexing performance.
2. Ability to not overwrite the existing document (based on listing_id, I
assume I can use overwrite=false flag to disable overwrite).
3. Not block the push indexing when delta indexing happens (push indexing
happens via UI, user should be able to search for the document pushed via UI
almost instantaneously). Delta processing might take more time to complete
indexing and I don't want the queue to wait until the batch processing is
complete.
4. Copy the updated collection for backup.

*More information on setup:
*We have 100 million records (around 6 stored fields / 12 indexed fields).
We are planning to have 5 cores (each with 20 million documents) with 5
replicas.
We will be always doing delta batch indexing.




--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Cloud-How-to-balance-Batch-and-Queue-indexing-tp4081169.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message