lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: no concurrent merging?
Date Wed, 10 Aug 2016 09:43:40 GMT
On Tue, Aug 9, 2016 at 7:14 PM, Yonik Seeley <yseeley@gmail.com> wrote:

But other than perhaps changing how long a DBQ takes to execute, it
> should be unrelated to the question of if other merges can proceed in
> parallel.
>
> A quick look at the lucene IndexWriter code says, no... Lucene DBQ
> processing cannot proceed in parallel.
> IndexWriter.mergeInit is synchronized (on IW).  The DBQ processing is
> called from there and thus anything else that needs the IW monitor
> will block.


Yeah, the impl Lucene has today is fully sync'd on IW while deletes are
being resolved (from Query or Term to docIDs).

I don't think there's a fundamental reason that it has to be, i.e. "it's
just software" and we could fix this (patches welcome!), but it's hairy.

I do think we would need a lock on each segment that's resolving its
deletes, or maybe even that could be made concurrent e.g. where a thread
can pull out one Query at a time and resolve it.

In any case, fixing it will likely be complex!

Mike McCandless

http://blog.mikemccandless.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message