lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <>
Subject Re: Lock-less commits
Date Fri, 25 Aug 2006 06:46:39 GMT

If i'm understanding this suggestion correctly, the main change in
observable behavior will be that actions performed by a "reader" will
never block or invalidate actions performed by a "writer" -- writers on
the other hand can still block eachother.

This seems like it might be the opposite of what most people would want:
that opening "reader" threads for doing searches need to be fast, and if a
writer thread has to wait a half second that's okay.

I also don't believe this would "solve" the NFS issues with regards to the
commit lock -- as i recall, the problem stems from NFS not being able to
garuntee transactional order of file operations (ie: i open the commit
lock file, i modify and close segments, i close/delete the commit file --
a remote NFS client might still see the orriginal segments file after the
commit file is deleted.  Your version file might suffer the same fate
(with reader clients seeing V1==V2 because the whole file is a second

: Date: Thu, 24 Aug 2006 23:22:56 -0700
: From: Doron Cohen <>
: Reply-To:
: To:
: Subject: Re: Lock-less commits
: I would like to discuss an additional approach, that requires small changes
: to current Lucene implementation. Here, the index version (currently in
: segments file) is maintained in a separate file, and is used to synchronize
: between readers and writers, without requiring readers to create/obtain any
: lock files, and without requiring readers to write anything to disk.
: - Index version would be maintained in a separate, dedicated Version file -
: (say .vsn) - one per index.
: - Version file contains two occurrences of the version number - V1 and V2.
: - In steady state, V1 == V2, but During update V1 == V2+1.
: - Every commit would:
:   - obtain a write lock (as today), to guarantee single writer at a time.
:   - increments V1 in that file, using RandomAccessFile API (RAF).
:     notice: now V1 != V2.
:   - do the commit work (merge, delete, whatever).
:   - increments V2 in that file, using RAF.
:     notice: now, again, V1 == V2.
:   - release the write lock (as today)
: - Every reader would read the version data in opposite order:
:   (1) read V2 from the version file, using RAF.
:   (2) read V1 from the version file, using RAF.
:   (3) if not V1==V2 wait some time, and try again (from step 1), until
: V1==V2, or timeout and fail.
:   (4) initialize reader data (read segment infos, open files).
:   (5) read again V2 then V1 using RAF.
:   (6) if not V2==V1 or they changed from steps 1 and 2, try again (from
: step 1), or timeout and fail.
: A few points to notice:
: - Using RAF protects from errors due to IO buffering.
: - Only tiny amount of version data is being read/written using RAF, so
: performance should not degrade.
: - Readers are not writing any data, so they are faster (A reader that does
: deleteDoc is a writer in this regard).
: - The opposite read/write order of RAF operations, i.e. writing V1 and then
: V2 by writer but reading V2 and then V1 by reader, protects from race
: conditions between readers and writers that otherwise might have caused
: reading corrupt data and concluding wrongly that the data is consistent
: while in fact it is not.
: - By using RAF and by the order of operations above, this scheme would work
: also for NFS (excluding the write lock mechanism which remains an issue in
: NFS).
: - For backward compatibility with current index structure the code can fall
: back to obtain a commit lock file if the .vsn file does not exist, and then
: create that .vsn file if it still does not exist.
: Regards,
: Doron
: ---------------------------------------------------------------------
: To unsubscribe, e-mail:
: For additional commands, e-mail:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message