lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liviu Matei <liviu...@gmail.com>
Subject Re: Issue with Lucene 3.6.1 and MMapDirectory
Date Mon, 19 May 2014 09:28:33 GMT
Also one more thing ... sorry forgot to add by using lsof I noticed deleted
index files that are still used by the application. Is this ok? Can't this
cause issues? The IndexReader trying to access an index file that was
deleted ? I suspect the deletion happens because of index merges during
indexing.

Thanks,
Liviu


On Mon, May 19, 2014 at 12:15 PM, Liviu Matei <liviumat@gmail.com> wrote:

> Thank you very much to all of you the answers.
> Uwe this is the strange thing that I am currently never closing the index
> reader and opening a new one from 8 to 8 hours and I am noticing that crash
> in indeed a highly concurrent environment.
> The indexes reside in a NFS file system. And the location is shared
> between multiple machines - on each machine running  multiple JVMs this is
> why I mentioned shared mount.
> Can you please tell me what parameters/settings should I check on the OS
> side? I did ulimit and it returns unlimited. I will check for "
> AlreadyClosedException" but I didn't saw that in the logs, but I will
> check again.
>
> 2. With the next release I am trying to do the close of IndexReader. I
> looked at SearchManager but what I am doing when doing a search I am also
> doing a indexing of the search content in order to search in it also in
> order to determine of score of the queries that I am constructing in the
> itself content. With SearchManager if I am correct I cannot do that but
> Clemens's approuch with ir.incRef(); should be ok also correct?
>
> Thank you very much again,
> Liviu
>
>
>
>
> On Fri, May 16, 2014 at 11:55 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
>
>> Hi,
>>
>> > Now if I don't
>> > close the old index reader I am noticing increases of virtual memory
>> with
>> > every new reindex reopen (this should not be an issue on 64 bit Linux
>> > correct - this is the configuration I am using and the indexes are on a
>> > shared mount NTFS file system ).
>>
>> This always brings a virtual memory leak on all platforms (also Linux).
>> In addition, files of older segments cannot be completely deleted anymore,
>> so it also consumes disk space.
>>
>> > Can you please tell me if all this corruption is caused by the fact
>> that I
>> > am not closing the old IndexReader. But if I close if given that it is
>> > share by multiple threads I will need to check each time before doing
>> the
>> > search if IndexReader is still open correct? Let's say in a thread I am
>> > reopening the IndexReader and in another thread I am afterwards reusing
>> the
>> > old one in that case I should do the check correct? Or is there a
>> smarter
>> > mechanism in place.
>>
>> It is the other way round: If you not close the IndexReader it cannot
>> crash (unless your JDK has a bug or somehow your filesystem [you mentioned
>> shared... what does this mean?] forcefully unmaps the index files), it only
>> happens if you close it! The issue here is: If you close the IndexReader
>> and another thread is currently running a query, the above can happen,
>> because the memory mapped buffer was forcefully unmapped by the
>> MMapDirectory. Since Lucene 3.6.0, Lucene tries its best to prevent this
>> crash from happening, but in high concurrency cases this may fail (because
>> of missing synchronization, which would kill performance):
>> http://issues.apache.org/jira/browse/LUCENE-3588
>>
>> In your case, in parallel to those crashes, you should also see
>> "AlreadyClosedException", which is the root of the problem. It is just
>> sometimes that the MMapDirectory code cannot correctly detect the already
>> closed and crashes.
>>
>> So forcefully reopening indexreaders and closing the old ones, while
>> queries are running is the wrong way to go. I would suggest to use
>> SearcherManager, which can keep track of the indexreaders correctly.
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
>> > Sent: Wednesday, May 14, 2014 7:53 PM
>> > To: java-user@lucene.apache.org
>> > Subject: AW: Issue with Lucene 3.6.1 and MMapDirectory
>> >
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message