lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Kohlschütter (JIRA) <>
Subject [jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache
Date Tue, 08 Dec 2009 19:02:18 GMT


Christian Kohlschütter commented on LUCENE-2135:

Honestly, please have a look at LUCENE-2133, I really think it is a good starting point to
solve all these problems. Could we perhaps merge the two issues (LUCENE-2133 and LUCENE-2135)?

A quick summary of LUCENE-2133:

The patch allows one or more IndexReaders to share common cache information (whatever this
is), stored in the same "IndexCache" instance. The IndexCache is designed to contain *any*
cacheable/volatile information that can be regenerated from the IndexReader.

For example: all clones of SegmentReader share the same SegmentReaderIndexCache with the original
instance, containing the ThreadLocals of the "core reader".
By default (for all IndexReader classes) the IndexCache provides access to the "IndexFieldCache"
(a non-static reimplementation of FieldCache).

To provide arbitrary cacheable objects we could now extend IndexCache by a simple HashMap
(it does not need to be a WeakHashMap, since the IndexCache is closed and purged as soon as
the original IndexReader is closed).

If you wish so, with the help of IndexCache we might even easily implement two different field
caches for the same IndexCache instance, one that changes with deleted docs and another one
that does not. Basically we may add any other kind of cache at a later point *without* touching
IndexReader again. To re-use Earwin Burrfoot statement from above, that would then *not* "complicate
IndexReader beyond that unholy mess it already is." 

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>                 Key: LUCENE-2135
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>         Attachments: LUCENE-2135.patch, LUCENE-2135.patch
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy
on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And,
apparently, WeakHashMap has a further impl detail that requires invoking one of its methods
for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long
as the sub-readers are truly closed (refCount dropped to 0).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message