lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Busch <busch...@gmail.com>
Subject Re: Question about aborting background merges
Date Mon, 03 Dec 2007 10:10:08 GMT
Michael McCandless wrote:

> Hmm ... looking at the code, I think we should also check whether a
> merge was aborted in mergeInit.  It looks like there is a window from
> when a merge is handed out until when it is inited such that if the
> writer is closed in that window it could result in incorrect re-use of a
> segment name.  Michael are you seeing such a case?
> 

Thanks for the explanation! I'm not seeing this particular issue.
However, a very weird thing happened in one of our stress tests. I
believe in that test a script would push a bunch of docs into the
IndexWriter, then close(false) the writer and open a new one immediately
afterwards. This test ran for quite a while without any problems (up to
8M docs). Then the weird thing happened: suddenly the index was totally
wiped out, meaning only the segments.gen and segments_1 were in the
index directory. There was no IOException in the log.

I must say that I didn't write the stress test and haven't seen it yet
either, I'm planning to take a look next week if I can. It looks like it
might be an application bug, such that it might open an IndexWriter with
create=true, because the segments file has the initial generation 1.

I started looking if the constructors of IndexWriter that automatically
try to determine if the index has to be created or not might have a
problem, maybe even in combination with aborted, but still running
background merge threads. I think the question is if
SegmentInfos.getCurrentSegmentGeneration() could ever falsely return -1,
but it doesn't look like this is possible.

I'm just really guessing here, it might not be a Lucene problem at all,
but at this moment I can't rule this possibility out.

-Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message