lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-753) Use NIO positional read to avoid synchronization in FSIndexInput
Date Wed, 02 Jul 2008 08:36:45 GMT


Michael McCandless commented on LUCENE-753:

SeparateFile (opening a new file descriptor per reader) is not a real implementation that
could be used... it's more of a theoretical maximum IMO. Also remember that you can't open
a new fd on demand since the file might already be deleted. We would need a real PooledClassicFile
implementation (like PooledPread).

True, we'd have to make a real pool, but I'd think we want the sync() to be on cloning and
not on every read.  I think Lucene's usage of the open files (clones are made & used up
quickly and closed) would work well with that approach.  I think at this point we should build
out an underlying pool and then test all of these different approaches under the pool.

And yes we cannot just open a new fd on demand if the file has been deleted.  But I'm thinking
that may not matter in practice.  Ie if the pool wants to open a new fd, it can attempt to
do so, and if the file was deleted it must then return a shared access wrapper to the fd it
already has open.  Large segments are where the contention will be and large segments are
not often deleted.  Plus people tend to open new readers if such a large change has taken
place to the index.

On non-windows it looks like ChannelPread is probably the right choice.. near max performance
and min fd usage

But on Linux I saw 44% speedup for serial=true case with 4 threads using SeparateFile vs ChannelPread,
which I was very surprised by.  But then again it's synthetic so it may not matter in real
Lucene searches.

> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>                 Key: LUCENE-753
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Yonik Seeley
>         Attachments:,,,,,,, FSIndexInput.patch, FSIndexInput.patch,
> As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying
> This could mitigate any MT performance drop caused by reducing the number of files in
the index format.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message