lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Vigdor (JIRA)" <>
Subject [jira] Commented: (LUCENE-831) Complete overhaul of FieldCache API/Implementation
Date Wed, 19 Nov 2008 22:10:44 GMT


Alex Vigdor commented on LUCENE-831:

Another useful feature that seems like it would be pretty easy to implement on top of this
patch is cache warming; that is, the ability to reopen an IndexReader and repopulate its caches
before making it "live".  The main thing missing is a Cache.getKeys() method which could be
used to discover what caches are already in use, in order to repopulate them after reopening
the IndexReader.  This cache warming could be performed externally to the IndexReader (by
calling getCacheData for each of the keys after reopening), or perhaps the reopen method could
be overloaded with a boolean "reloadCaches" to perform this in the same method call.

The rationale for this I hope is clear; my application, like many I'm sure, keeps a single
IndexReader for handling searches, and in a separate thread from search request handling commits
writes and reopens the IndexReader before replacing the IndexReader reference for new searches
(reference counting is then used to eventually close the old reader instances).  It would
be ideal for that separate thread to also bear the expense of cache warming; even with this
patch against our 4 GB indices, along with -Duse.object.array.sort=true,  a search request
coming immediately after reopening the index will pause 20-40 seconds while the caches refill.
 Preferably that could be done in the background and request handling threads would never
be slowed down.

> Complete overhaul of FieldCache API/Implementation
> --------------------------------------------------
>                 Key: LUCENE-831
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>            Reporter: Hoss Man
>             Fix For: 3.0
>         Attachments: fieldcache-overhaul.032208.diff, fieldcache-overhaul.diff, fieldcache-overhaul.diff,
LUCENE-831.03.28.2008.diff, LUCENE-831.03.30.2008.diff, LUCENE-831.03.31.2008.diff, LUCENE-831.patch,
LUCENE-831.patch, LUCENE-831.patch
> Motivation:
> 1) Complete overhaul the API/implementation of "FieldCache" type things...
>     a) eliminate global static map keyed on IndexReader (thus
>         eliminating synch block between completley independent IndexReaders)
>     b) allow more customization of cache management (ie: use 
>         expiration/replacement strategies, disk backed caches, etc)
>     c) allow people to define custom cache data logic (ie: custom
>         parsers, complex datatypes, etc... anything tied to a reader)
>     d) allow people to inspect what's in a cache (list of CacheKeys) for
>         an IndexReader so a new IndexReader can be likewise warmed. 
>     e) Lend support for smarter cache management if/when
>         IndexReader.reopen is added (merging of cached data from subReaders).
> 2) Provide backwards compatibility to support existing FieldCache API with
>     the new implementation, so there is no redundent caching as client code
>     migrades to new API.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message