lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "G Fernandes (JIRA)" <>
Subject [jira] [Commented] (LUCENE-5541) FileExistsCachingDirectory, to work around unreliable File.exists
Date Wed, 17 Sep 2014 21:33:34 GMT


G Fernandes commented on LUCENE-5541:

In the Infinispan Directory case, because of a tricky race condition, our File.exists() was
not totally reliable as well when under load: so we were seeing exactly the same behaviour
and stack trace described in this issue. Fortunately, differently from the FSDirectory impl,
we were able to fix the race condition and the problem was solved, since our File.exists actually
started to behave correctly. Fun :)

> FileExistsCachingDirectory, to work around unreliable File.exists
> -----------------------------------------------------------------
>                 Key: LUCENE-5541
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/store
>            Reporter: Michael McCandless
>         Attachments: LUCENE-5541.patch
> File.exists is a dangerous method in Java, because if there is a
> low-level IOException (permission denied, out of file handles, etc.)
> the method can return false when it should return true.
> Fortunately, as of Lucene 4.x, we rely much less on File.exists,
> because we track which files the codec components created, and we know
> those files then exist.
> But, unfortunately, going from 3.0.x to 3.6.x, we increased our
> reliance on File.exists, e.g. when creating CFS we check File.exists
> on each sub-file before trying to add it, and I have a customer
> corruption case where apparently a transient low level IOE caused
> File.exists to incorrectly return false for one of the sub-files.  It
> results in corruption like this:
> {noformat}
> No sub-file with id .fnm found (fileName=_1u7.cfs files:
[.tis, .tii, .frq, .prx, .fdt, .nrm, .fdx])
>       org.apache.lucene.index.CompoundFileReader.openInput(
>       org.apache.lucene.index.CompoundFileReader.openInput(
>       org.apache.lucene.index.FieldInfos.<init>(
>       org.apache.lucene.index.IndexWriter.getFieldInfos(
>       org.apache.lucene.index.IndexWriter.getCurrentFieldInfos(
>       org.apache.lucene.index.IndexWriter.<init>(
> {noformat}
> I think typically local file systems don't often hit such low level
> errors, but if you have an index on a remote filesystem, where network
> hiccups can cause problems, it's more likely.
> As a simple workaround, I created a basic Directory delegator that
> holds a Set of all created but not deleted files, and short-circuits
> fileExists to return true if the file is in that set.
> I don't plan to commit this: we aren't doing bug-fix releases on
> 3.6.x anymore (it's very old by now), and this problem is already
> "fixed" in 4.x (by reducing our reliance on File.exists), but I wanted
> to post the code here in case others hit it.  It looks like it was hit
> e.g. and

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message