lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuck Williams (JIRA)" <>
Subject [jira] Commented: (LUCENE-555) Index Corruption
Date Mon, 24 Apr 2006 21:40:08 GMT
    [ ] 

Chuck Williams commented on LUCENE-555:

I'm surprised that an optimize led to a corrupt index.  Other than the non-atomic rename problem,
there isn't anything else in Lucene that should lead to corruption.  Even a failure in rename
is recoverable, since some correct version of the segments file is always available.

The one recovery issue I've encountered is the buffering of recently indexed documents in
memory.  I have a selective journaling mechanism that saves just as many documents as might
ever be buffered in RAM.  This mechanism supports several logging modes:  complete information,
just stored fields wth the ability to compute the others, and just keys with the ability to
retrieve externally.  A mechanism like this could be extended to support unlimited journaling
if you really want it.

If the optimize of an index leaves it corrupted, this is a bug that should be fixed.  If Lucene
is robust in the sense that it doesn't corrupt the index, as it is designed to be now, I think
this is sufficient.

Optional journaling facilties would be a nice add-on feature.  It is somethat that is not
too hard for applications to create now.  The mechanism I use is bundled with a number of
other useful services, including updating the index by modifying field values of selected
documents, managing synchronization of delete, write and update, managing the periodic refreshing
of the reader used for search, etc.  If I can get agreement from my Company, the I'll contribute
some of all of this.  Maybe it would be of help.

> Index Corruption
> ----------------
>          Key: LUCENE-555
>          URL:
>      Project: Lucene - Java
>         Type: Bug

>   Components: Index
>     Versions: 1.9
>  Environment: Linux FC4, Java 1.4.9
>     Reporter: dan
>     Priority: Critical

> Index Corruption
> >>>>>>>>> output
> ../_aki.fnm (No such file or directory)
>         at Method)
>         at<init>(
>         at$Descriptor.<init>(
>         at<init>(
>         at
>         at org.apache.lucene.index.FieldInfos.<init>(
>         at org.apache.lucene.index.SegmentReader.initialize(
>         at org.apache.lucene.index.SegmentReader.get(
>         at org.apache.lucene.index.SegmentReader.get(
>         at org.apache.lucene.index.IndexWriter.mergeSegments(
>         at org.apache.lucene.index.IndexWriter.mergeSegments(
>         at org.apache.lucene.index.IndexWriter.optimize(
> >>>>>>>>> input
> - I open an index, I read, I write, I optimize, and eventually the above happens. The
index is unusable.
> - This has happened to me somewhere between 20 and 30 times now - on indexes of different
shapes and sizes.
> - I don't know the reason. But, the following requirement applies regardless.
> >>>>>>>>> requirement
> - Like all modern database programs, there has to be a way to repair an index. Period.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message