lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: Lock-less commits
Date Sat, 26 Aug 2006 11:07:17 GMT
Doron Cohen wrote:
>> In my local changes (using numbered files) for lock-less commits, I've
>> implemented Yonik's suggestsion of opening segments in reverse order,
>> and this has definitely reduced the number of "retries" that the
>> searchers hit on opening the index.  Even in highly interactive
>> searching (open searcher, do one search, close searcher, repeat) the
>> retry rate is low.
> In this highly interactive search scenario is it true that every opened
> searcher needs a directory listing? - If so is this a possible performance
> hit for the searchers, similar to discussion in this thread for writers.
> But we should worry more for searchers...  In this case, how about
> maintaining a separate version-file (as discussed) for allowing new
> searchers/readers to easily detect the most current version that is stable
> and should be used, without needing a directory listing. I believe that
> NFS-wise, it should be as safe to use as a directory listing (more on NFS
> below).

Yes in my current implementation I do a dir listing to locate the
current segments file.

>> The big problem is the client's caching.  I've seen cases in my own
>> testing where the NFS cache on one machine remains stale for quite some
>> time (seconds) before "seeing" changes to a file on a server.  I think
>> instead relying on a newly created file with the numbered approach (ie
>> never before used file name) will avoid the risk that a client-side
>> cache is presenting stale (or delayed) contents of a file.
> I can't see why relying on files newly created is safer than relying on
> files that were updated. In other words, I think we are ok with stales -
> e.g. if the writer wrote 5 new files, and then wrote a 6th new file - the
> appropriate segments file, it seems the numbered-files approach too is
> counting on that the first 5 written files are visible to any client once
> the 6th file is visible, otherwise readers might not find the files they
> are looking for (the segment-infos just read from the 6th file). I
> understand "staling" as - "although the 6th file was already written, a
> reader might not yet see it yet, it would only see the first 5 files". But
> this is not a problem, because the reader would attempt to use the previous
> version. If that would fail, the reader would retry and now probably get
> the newer version.
> So it seems that the numbered-files scheme also relies on proper order of
> operations, or I might misunderstand something here. Under this same
> "visibility order assumption", I think the version-file should be safe to
> use in NSF.

OK I think there are two issues here:

   * Is using a newly created file actually safer than re-using a

     I believe it should be: by using a newly created filename, there
     is no risk that this file's contents is in the local (possibly
     stale) filesystem cache.  Ie the filesystem is forced to go to the
     server to retrieve the contents of the file.  Beyond NFS, I would
     think we get ourselves into less trouble with filesystem caching
     by never re-using a filename.

     One other important reason: users have reported errors in renaming -> segments (and deletable, but that file is no
     longer used with my changes) on Windows.  This is probably due to
     the fact that a reader is reading the segments at that moment.
     While will could insert a "sleep; try again" loop into the writer,
     but that's adding possibly unnecessary latency.

   * Relying on "order of operations": yes, I agree, the numbered file
     approach does rely on a limited use-case of this (see below).
     However, so does the current Lucene (with its current locking or
     even once we get native locking implemented).

     So in this regard numbered files is no worse than the current
     Lucene -- "first do no harm" :)

     The numbered file locking relies on a limited use case "creation
     of new files in a single directory".  I'm certainly no NFS expert
     (but I'm rapidly getting tempted to buy NFS Illustrated!!), but
     looking at how the NSF client/server protocol works, I would be
     surprised if this specific use case fails to preserve of
     order-of-file-creates.  An NFS client client can't know about a
     new file X until it contacts the server to do a READDIR or
     READDIRPLUS, and at the point that it does so, it should get the
     then-current current list of files in the directory.

     Whereas, the current Lucene does rely on a more powerful "order of
     operations" involving fresh contents of the "segments" file
     against existence/freshness of the separate segments files.  This
     could indeed be the source of the FileNotFound exceptions users
     see over NFS (or, it could also be that is not actually atomic over NFS).

     I agree that "order of operations" in general, if you span
     multiple directories and you're including in the fresh contents of
     re-used filenames, is very likely not guaranteed by NFS or other
     remote filesystems: it's just very costly to do truly correctly.

 > Under this no-order assumption, it is also possible that a file 
exists but
 > only part of it is available for the reader - to defend against this the
 > reader should be able to verify that the segments file is complete, and
 > that each of the other files is complete - using the file length or so,
 > right?

Unfortunately, relying on accurate file sizes through NFS caching is
dangerous.  In my local mods I take the same approach as before, ie,
write to a .new file and then rename it.  This way the shear existence
of a segments_N file means it's now complete.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message