lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alon Muchnick <a...@datonics.com>
Subject potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .
Date Thu, 03 Jan 2013 00:29:02 GMT
hi MIke,
thank you very much for replying .

i am not sure when the doClose() method of the SegementReader will be
called  ,
but i think you are correct regarding the SegmentReader still being live ,
we have an singleton object with a hash map of several RamDirectory indexes
and their IndexReader, IndexSearcher and IndexWriter objects ,
this object is initialized when the system starts and only closed when the
system shuts down .

the upgrade to tomcat 7 did increase the amount of live threads (due to
changes in tomcat's thread pool behavior ) and could explain more RAM being
used , but the scenario i am seeing is that even after the application had
stopped receiving new connection ,the thread pool is reduced to minimum
size and GC had been run , the are still references to the closed threads
as the keys in the weak hash map.

am i wrong to assume that under the above scenario the weakhash map should
have been truncated to hold only keys for the remaining live threads in the
the thread pool ? or will this never happen because there will be no more
additonal search requests thus the countUntilPurge in CloseableThreadLocal
will never reach 0 and the purge method will be never called?


Just to be clear regarding the sevirity of this , when the system comes up
the 6 RAMDirectory Lucune indexes are about 30 MG , after a few hours
,when we sample it after the thread pool returned to minimum size and a GC
had been run the Lucune indexes take 500MG .

i guess its not clear to me when will the memory in  CloseableThreadLocals
object will be release when working with a thread pool  ?


Thanks in advanced

Alon


On Wed, Jan 2, 2013 at 10:46 PM, Michael McCandless <
lucene@mikemccandless.com <javascript:_e({}, 'cvml',
'lucene@mikemccandless.com');>> wrote:

> CloseableThreadLocal.hardRefs is still a live reference then it
> means the SegmentReader hasn't been closed yet, ie I think the chain
> you show below is from a still-open SegmentReader?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message