lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley (JIRA)" <>
Subject [jira] Commented: (LUCENE-753) Use NIO positional read to avoid synchronization in FSIndexInput
Date Wed, 02 Jul 2008 13:21:45 GMT


Yonik Seeley commented on LUCENE-753:

bq. (clones are made & used up quickly and closed) 

IIRC, clones are often not closed at all.
And for term expanding queries, you can get a *lot* of them all at once.

bq. And yes we cannot just open a new fd on demand if the file has been deleted. But I'm thinking
that may not matter in practice. Ie if the pool wants to open a new fd, it can attempt to
do so, and if the file was deleted it must then return a shared access wrapper to the fd it
already has open.

At first blush, sounds a bit too complex for the benefits.
- one would have to reserve the last fd for synchronized access... can't really hand it out
for unsynchronized exclusive access and then go and share it later.
- the shared access should use pread... not seek+read

bq. But on Linux I saw 44% speedup for serial=true case with 4 threads using SeparateFile
vs ChannelPread, which I was very surprised by.

In the serial case, there are half the system calls (no seek).  When both implementations
have a single single system call, all the extra code and complexity that Sun threw into FileChannel
comes into play.  Compare that with which drops down to a native call
and presumably just the read with little overhead.  I wish Sun would just add a
with a file position.

If access will be truly serial sometimes, larger buffers would help with that larger read()
setup cost. 

> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>                 Key: LUCENE-753
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Yonik Seeley
>         Attachments:,,,,,,, FSIndexInput.patch, FSIndexInput.patch,
lucene-753.patch, lucene-753.patch
> As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying
> This could mitigate any MT performance drop caused by reducing the number of files in
the index format.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message