lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liviu Matei <liviu...@gmail.com>
Subject Issue with Lucene 3.6.1 and MMapDirectory
Date Wed, 14 May 2014 09:05:33 GMT
Hi,

I am encountering the following issue with Lucene 3.6.1 if you could let me
know if I am doing something wrong / there is a mistake I am making it
would be great.

In order to improve the performance of the application that I am working at
I went to the approach of reusing the IndexReader and reopening it every 8
hours in order to get the latest changes. (IndexReader is declared as a
global static variable). The search method is called from multiple threads
in parallel so the index reader is shared between threads. Now if I don't
close the old index reader I am noticing increases of virtual memory with
every new reindex reopen (this should not be an issue on 64 bit Linux
correct - this is the configuration I am using and the indexes are on a
shared mount NTFS file system ).
   Also from time to times I noticed JVM crasches with the following stack:
 Thread Stack Trace:
    at memcpy+160()@0x381aa7b060
    -- Java stack --
    at java/nio/DirectByteBuffer.get(DirectByteBuffer.java:294)[optimized]
    at
org/apache/lucene/store/MMapDirectory$MMapIndexInput.readBytes(MMapDirectory.java:298)[optimized]
    at org/apache/lucene/store/DataInput.readBytes(DataInput.java:72)
    at
org/apache/lucene/index/CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:275)[optimized]
    at
org/apache/lucene/store/BufferedIndexInput.refill(BufferedIndexInput.java:270)[optimized]
    at
org/apache/lucene/store/BufferedIndexInput.readByte(BufferedIndexInput.java:40)[inlined]
    at
org/apache/lucene/store/DataInput.readVInt(DataInput.java:107)[inlined]
    at
org/apache/lucene/store/BufferedIndexInput.readVInt(BufferedIndexInput.java:217)[optimized]
    at org/apache/lucene/index/FieldsReader.doc(FieldsReader.java:235)
    at
org/apache/lucene/index/SegmentReader.document(SegmentReader.java:492)
    at
org/apache/lucene/index/DirectoryReader.document(DirectoryReader.java:568)
    at org/apache/lucene/index/MultiReader.document(MultiReader.java:252)
    at org/apache/lucene/index/IndexReader.document(IndexReader.java:1138)
    at
org/apache/lucene/search/IndexSearcher.doc(IndexSearcher.java:258)[inlined]


Can you please tell me if all this corruption is caused by the fact that I
am not closing the old IndexReader. But if I close if given that it is
share by multiple threads I will need to check each time before doing the
search if IndexReader is still open correct? Let's say in a thread I am
reopening the IndexReader and in another thread I am afterwards reusing the
old one in that case I should do the check correct? Or is there a smarter
mechanism in place.

Any help with this would be more than welcome.


Thank you very much and best regards,
Liviu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message