lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: CorruptIndexException when opening Index during first commit
Date Fri, 17 May 2013 11:13:43 GMT
On Thu, May 16, 2013 at 2:59 PM, Geoff Cooney <cooney.geoff@gmail.com> wrote:
> Thanks for the response, Mike.
>
> If I understand correctly, the problem was incorrectly identifying a large
> corrupted index as a non-existant index?

Actually, a large healthy index as non-existent (because of file
descriptor exhaustion).

> It seems like you'd really want
> an index with first-commit in progress to behave like an index with zero
> documents, as opposed to a non-existant index.  That is, if segments_1 is
> corrupt, open an empty index.  This seems more analagous to how lucene
> treats pre-committed segments_n files after the first.  I don't know how
> difficult it would be to implement, in practice.

Hmm, I think first-commit in progress should really act like no index
exists yet.  If we instead act
like a 0-docs index that can mask true corruption.

The problem is we can't reliably differentiate commit-in-progress from
a corrupt first commit...

> This isn't a major issue for us either way.  We can work around it by
> having our searcher code check for the existence of a segments.gen and just
> return zero hits by default.  Since this is a read path only, we don't need
> to worry about incorrectly overwriting the index.

Ahh sneaky but that should work (unless IW changes its index format
wrt segments.gen).

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message