lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Earwin Burrfoot (JIRA)" <>
Subject [jira] Commented: (LUCENE-1658) Absorb NIOFSDirectory into FSDirectory
Date Tue, 26 May 2009 13:07:45 GMT


Earwin Burrfoot commented on LUCENE-1658:

bq. Wait, are you saying Win 64 bit has problems w/ MMapDir? (I thought you just said Win
32 bit, above).
I have no lucene experience with Win64, and so I extrapolated from 32bit (as I felt it was
more of a Windows issue than anything else). Would be nice if someone actually tries.
If it works, then the rule sounds like - MMap for all 64bit systems, NIO for 32bit non-win,
Simple for 32bit win.

bq. Is MMapDir fine w/ concurrency?
It's cool with it if you have enough memory (no frequent paging occurs). I'm not sure about
NIOFS vs MMap on memory-constrained systems, if you're competing for disk cache.

bq. Which way do you think "prime all bytes up front on open" should default?
This has a side-effect of pushing previous mmaps out of memory if you're memory-constrained.
Thus, under certain usage conditions (frequent merging, or something like that + low memory)
this feature theoretically could be a disadvantage.
For me it works well enough to be hardcoded to true :)

If anybody's interested, I can also repost an alternative for MMapD - MemoryMirroredD, which
wraps any given D and explicitly preloads files in non-chunked byte arrays. It's a bit faster
than MMapD and MemoryD (for reading), but has certain disadvantages, like stressing your GC
and throwing OOM on merges/optimize if you don't have enough heap (unlike MMapD that silently
swaps out).

> Absorb NIOFSDirectory into FSDirectory
> --------------------------------------
>                 Key: LUCENE-1658
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: LUCENE-1658.patch, LUCENE-1658.patch
> I think whether one uses* vs java.nio.* or eventually
> java.nio2.*, or some other means, is an under-the-hood implementation
> detail of FSDirectory and doesn't merit a whole separate class.
> I think FSDirectory should be the core class one uses when one's index
> is in the filesystem.
> So, I'd like to deprecate NIOFSDirectory, absorbing it into
> FSDirectory, and add a setting "useNIO" to FSDirectory.  It should
> default to "true" for non-Windows OSs, because it gives far better
> concurrent performance on all platforms but Windows (due to known Sun
> JRE issue

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message