lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <>
Subject Re: Question about aborting background merges
Date Mon, 03 Dec 2007 10:36:55 GMT

"Michael Busch" <> wrote:
> Michael McCandless wrote:
> > Hmm ... looking at the code, I think we should also check whether a
> > merge was aborted in mergeInit.  It looks like there is a window from
> > when a merge is handed out until when it is inited such that if the
> > writer is closed in that window it could result in incorrect re-use of a
> > segment name.  Michael are you seeing such a case?
> > 
> Thanks for the explanation! I'm not seeing this particular issue.
> However, a very weird thing happened in one of our stress tests. I
> believe in that test a script would push a bunch of docs into the
> IndexWriter, then close(false) the writer and open a new one immediately
> afterwards. This test ran for quite a while without any problems (up to
> 8M docs). Then the weird thing happened: suddenly the index was totally
> wiped out, meaning only the segments.gen and segments_1 were in the
> index directory. There was no IOException in the log.
> I must say that I didn't write the stress test and haven't seen it yet
> either, I'm planning to take a look next week if I can. It looks like it
> might be an application bug, such that it might open an IndexWriter with
> create=true, because the segments file has the initial generation 1.
> I started looking if the constructors of IndexWriter that automatically
> try to determine if the index has to be created or not might have a
> problem, maybe even in combination with aborted, but still running
> background merge threads. I think the question is if
> SegmentInfos.getCurrentSegmentGeneration() could ever falsely return -1,
> but it doesn't look like this is possible.
> I'm just really guessing here, it might not be a Lucene problem at all,
> but at this moment I can't rule this possibility out.

Well that certainly sounds spooky!  You mean this test is using the
one of IndexWriter's ctors that relies on IndexReader.indexExists() to
decide whether to pass create=true?

Keep up posted!


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message