lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-753) Use NIO positional read to avoid synchronization in FSIndexInput
Date Wed, 02 Jul 2008 13:21:45 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609916#action_12609916
] 

Yonik Seeley commented on LUCENE-753:
-------------------------------------

bq. (clones are made & used up quickly and closed) 

IIRC, clones are often not closed at all.
And for term expanding queries, you can get a *lot* of them all at once.

bq. And yes we cannot just open a new fd on demand if the file has been deleted. But I'm thinking
that may not matter in practice. Ie if the pool wants to open a new fd, it can attempt to
do so, and if the file was deleted it must then return a shared access wrapper to the fd it
already has open.

At first blush, sounds a bit too complex for the benefits.
- one would have to reserve the last fd for synchronized access... can't really hand it out
for unsynchronized exclusive access and then go and share it later.
- the shared access should use pread... not seek+read

bq. But on Linux I saw 44% speedup for serial=true case with 4 threads using SeparateFile
vs ChannelPread, which I was very surprised by.

In the serial case, there are half the system calls (no seek).  When both implementations
have a single single system call, all the extra code and complexity that Sun threw into FileChannel
comes into play.  Compare that with RandomAccessFile.read() which drops down to a native call
and presumably just the read with little overhead.  I wish Sun would just add a RandomAccessFile.read
with a file position.

If access will be truly serial sometimes, larger buffers would help with that larger read()
setup cost. 


> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>
>                 Key: LUCENE-753
>                 URL: https://issues.apache.org/jira/browse/LUCENE-753
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Yonik Seeley
>         Attachments: FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java,
FileReadTest.java, FileReadTest.java, FileReadTest.java, FSIndexInput.patch, FSIndexInput.patch,
lucene-753.patch, lucene-753.patch
>
>
> As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying
file.
> This could mitigate any MT performance drop caused by reducing the number of files in
the index format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message