lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3237) FSDirectory.fsync() may not work properly
Date Thu, 10 Apr 2014 09:54:15 GMT


Simon Willnauer commented on LUCENE-3237:

Hey mike, thanks for reopening this. I like the patch since it fixes multiple issues. 

 * I like the fact that we get rid of the general unsynced files stuff in Directory.
 * given the last point we move it in the right place inside IW that is where it should be
 * the problem that the current patch has is that is holds on to the buffers in BufferedIndexOutput.
I think we need to work around this here are a couple of ideas:
  **  introduce a SyncHandle class that we can pull from IndexOutput that allows to close
the IndexOutput but lets you fsync after the fact
  ** this handle can be refcounted internally and we just decrement the count on IndexOutput#close()
as well as on SyncHandle#close() 
  ** we can just hold on to the SyncHandle until we need to sync in IW 
  ** since this will basically close the underlying FD later we might want to think about
size-bounding the number of unsynced files and maybe let indexing threads fsync them concurrently?
maybe something we can do later.
  ** if we know we flush for commit we can already fsync directly which might safe resources
/ time since it might be concurrent

just a couple of ideas....

> FSDirectory.fsync() may not work properly
> -----------------------------------------
>                 Key: LUCENE-3237
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/store
>            Reporter: Shai Erera
>         Attachments: LUCENE-3237.patch
> Spinoff from LUCENE-3230. FSDirectory.fsync() opens a new RAF, sync() its FileDescriptor
and closes RAF. It is not clear that this syncs whatever was written to the file by other
FileDescriptors. It would be better if we do this operation on the actual RAF/FileOS which
wrote the data. We can add sync() to IndexOutput and FSIndexOutput will do that.
> Directory-wise, we should stop syncing on file names, and instead sync on the IOs that
performed the write operations.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message