lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-702) Disk full during addIndexes(Directory[]) can corrupt index
Date Tue, 07 Nov 2006 23:20:53 GMT
    [ ] 
Michael McCandless commented on LUCENE-702:

I think we should try to make all of the addIndexes calls (and more
generally any call to Lucene) "transactional".  Meaning, if the call
is aborted (machine crashes, disk full, jvm killed, neutrino hits CPU,
etc.) then your index just "rolls back" to where it was at the start
of the call.  Ie, it is consistent and none of the incoming documents
were added.

This way your index is fine after the crash, and, you can fix the
cause of the crash and re-run the addIndexes call and you won't get
duplicate documents.

To achieve this, each of the three addIndexes methods would need to 1)
not commit a new segments file until the end, and 2) not delete any
segments referenced by the initial segments file (segmentInfos) until
the end.

We have three methods now for addIndexes:

  * For addIndexes(IndexReader[]): this method is I think already
    transactional.  We create a merger, add all readers to it, do the
    merge, and only at the end commit the new segments file & remove
    old segments.

  * For addIndexes(Directory[]): this method can currently corrupt the
    index if aborted.  However, because all merging is done only on
    the newly added segments, I think the fix is simply to not commit
    the new segments file until the end?

  * For addIndexesNoOptimize(Directory[]): this method can also
    currently corrupt the index if aborted.  To fix this I think we
    need to not only prevent committing a new segments file until the
    end, but also to prevent deletion of any segments in the original
    segments file.  This is because it's able (I think?) to merge
    both old and new segments in its step 3.  This would normally
    result in deleting those old segments that were merged.

    Note that this will increase the temporary disk usage used during
    the call, because old segments must remain on disk even if they
    have been merged, but I think this is the right tradeoff
    (transactional vs temporary disk usage)?

Also note that we would need the fixes from lockless LUCENE-701 to
properly delete orphan'd segments after an abort.  Without the
IndexFileDeleter I think they would stick around indefinitely.

Does this approach sound reasonable/right?  Any feedback?

> Disk full during addIndexes(Directory[]) can corrupt index
> ----------------------------------------------------------
>                 Key: LUCENE-702
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.1
>            Reporter: Michael McCandless
>         Assigned To: Michael McCandless
> This is a spinoff of LUCENE-555
> If the disk fills up during this call then the committed segments file can reference
segments that were not written.  Then the whole index becomes unusable.
> Does anyone know of any other cases where disk full could corrupt the index?
> I think disk full should worse lose the documents that were "in flight" at the time.
 It shouldn't corrupt the index.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message