lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-753) Use NIO positional read to avoid synchronization in FSIndexInput
Date Wed, 02 Jul 2008 18:01:45 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609990#action_12609990
] 

Michael McCandless commented on LUCENE-753:
-------------------------------------------

bq. And for term expanding queries, you can get a lot of them all at once.

Right but that'd all be under one thread right?  The pool would always give the same RandomAccessFile
(private or shared) for the same filename X thread.

bq. one would have to reserve the last fd for synchronized access... can't really hand it
out for unsynchronized exclusive access and then go and share it later.

Well, I think you'd hand it out first, as a shared file (so you reserve the right to hand
it out again, later).  If other threads come along you would open a new one (if you are under
the budget) and loan it to them privately (so no syncing during read).  I think sync'ing with
no contention (the first shared file we hand out) should be OK performance in modern JVMs.

bq. the shared access should use pread... not seek+read

But not on Windows...

bq. At first blush, sounds a bit too complex for the benefits.

Yeah I'm on the fence too ... but this lack of concurrency hurts our search performance. 
It's ashame users have to resort to multiple IndexReaders.  Though it still remains to be
seen how much the pool or pread approaches really improve end to end search performance (vs
other bottlenecks like IndexReader.isDeleted).

Windows is an important platform and doing the pool approach, vs leaving Windows with classic
if we do pread approach, lets us have good concurrency on Windows too.

> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>
>                 Key: LUCENE-753
>                 URL: https://issues.apache.org/jira/browse/LUCENE-753
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Yonik Seeley
>         Attachments: FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java,
FileReadTest.java, FileReadTest.java, FileReadTest.java, FSIndexInput.patch, FSIndexInput.patch,
lucene-753.patch, lucene-753.patch
>
>
> As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying
file.
> This could mitigate any MT performance drop caused by reducing the number of files in
the index format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message