lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chuck Williams <>
Subject Re: Fwd: How to combine results from several indices
Date Mon, 12 Jun 2006 20:55:37 GMT

I've contributed a version of ParallelWriter that takes a middle
ground.  ParallelWriter.addDocument() is synchronized, but the
underlying sub-index writes are done in parallel.  It would be possible
to allow ParallelWriter.addDocument() itself to be multi-threaded, but
the synchronization and recovery get more complex.  The basic idea would
be to have a thread with a work queue for each sub-index.  I may look at
this later, or if you enhance this, please submit your version.

Hope this helps,


Chuck Williams wrote on 06/12/2006 09:05 AM:
> Hi Wu,
> The simplest solution is to synchronize calls to a
> ParallelWriter.addDocument() method that calls IndexWriter.addDocument()
> for each sub-index.  This will work assuming there are no exceptions and
> assuming you never refresh your IndexReader within
> ParallelWriter.addDocument().  If exceptions occur writing one of the
> sub-indexes, then you need to recover them.  The best approach I've
> found is to delete the unequal final subdocuments and optimize all the
> subindexes to restore equal doc ids.
> This approach has the consequence of single-threading all index
> writing.  I'm working on a solution to avoid this, but it may require
> deeper integration into the higher level IndexManager mechanism (which
> does reader reopening, journaling, recovery, and a lot of other things).
> If you can get by with single threading, I have a ParallelWriter class
> now that I could contribute.  If not, I'm considering the more general
> solution now, but will only be able to contribute it if it can be kept
> separate from the much larger IndexManager mechanism (which is more
> specific to my app and thus not likely a fit for your app anyway).
> Chuck
> wu fox wrote on 06/12/2006 02:43 AM:
>> Hi Chuck:
>>  I am still looking forward to a solution which ensure to to meet the
>> constraints of
>> ParallelReader so that I can use it for my seach programm. I have
>> tried a lot of methods but none of them
>> is good enough for me because of obvious
>> bugs. Can you help me? thanks in advance
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message