lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Elschot <>
Subject Using MMapDirectory fails TestCompoundFile; MMapDirectory for huge indexes
Date Thu, 30 Sep 2004 19:45:35 GMT
On Wednesday 29 September 2004 00:00, Doug Cutting wrote:
> wrote:
> >   Added:       src/java/org/apache/lucene/store
> >   Log:
> >   Add an nio mmap based Directory implementation.
> For my simple benchmarks this is somewhat slower than the classic
> FSDirectory, but I thought it was still worth having.  It should use
> less memory when there are lots of query terms, since it does not need
> to allocate a new buffer per term and the mmapped data can be shared.
> This may be good for folks who, e.g., use lots of wildcards.  It also
> should, in theory, someday be faster.  One downside is that it cannot
> handle indexes with files larger than 2^31 bytes.

I'm working on a memory mapped directory that uses multiple buffers
for large files.

While trying some test runs I found that the current version fails a test:

    [junit] Testsuite: org.apache.lucene.index.TestCompoundFile
    [junit] Tests run: 9, Failures: 1, Errors: 0, Time elapsed: 4.238 sec

    [junit] Testcase: testClonedStreamsClosing(org.apache.lucene.index.TestCompoundFile):
    [junit] null
    [junit] junit.framework.AssertionFailedError
    [junit] 	at org.apache.lucene.index.TestCompoundFile.testClonedStreamsClosing(
    [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke(
    [junit] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(

As indicated in the source I have added a System property to the jvm
to use MMapDirectory with this is line 261 in build.xml for the junit
      <jvmarg value=""/>

I'm testing the version with multiple buffers using a smaller maximum
buffer size (1024 * 128), and it does this test in the same way.
I have not yet looked into TestCompoundFile. When it is a good test
case for this, I'll submit the multibuffer version as an enhancement.

Paul Elschot.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message