lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Heisey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-10015) Remove strong reference to Field Cache key (optional) so that GC can release some Field Cache entries when Solr is under memory pressure
Date Fri, 20 Jan 2017 21:58:26 GMT

    [ https://issues.apache.org/jira/browse/SOLR-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832469#comment-15832469
] 

Shawn Heisey commented on SOLR-10015:
-------------------------------------

I may be showing my ignorance here, but I have a question.

Would this prune individual items from within a larger field cache entry, or prune an entire
set of items at once?  The latter sounds OK, but the former seems like a very bad idea.  If
one of those large entries in the cache were to have a percentage of its elements removed
by GC, I think that search results would be incorrect.

I don't know enough about Lucene internals to be able to answer my question by looking at
the patch.

Assuming my worry above is unfounded, I think this is only likely to delay OOM instead of
preventing it, and even if it did prevent it, a low memory situation would probably involve
constant eviction (by GC) and recreation (by queries) of FC entries, which I would expect
to have strong performance implications.


> Remove strong reference to Field Cache key (optional) so that GC can release some Field
Cache entries when Solr is under memory pressure
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-10015
>                 URL: https://issues.apache.org/jira/browse/SOLR-10015
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Michael Sun
>         Attachments: SOLR-10015-prototype.patch
>
>
> In current Field Cache (FC) implementation, a WeakHashMap is used, supposedly designed
to allow GC to release some Field Cache entries when Solr is under memory pressure. 
> However, in practice, FC entry releasing seldom happens. Even worse, sometime Solr goes
OOM and heap dump shows large amount of memory is actually used by FC. It's a sign that GC
is not able to release FC entries even WeakHashMap is used.
> The reason is that FC is using SegmentCoreReaders as the key to the WeakHashMap. However,
SegmentCoreReaders is usually strong referenced by SegmentReader. A strong reference would
prevent GC to release the key and therefore the value. Therefore GC can't release entries
in FC's WeakHashMap. 
> The JIRA is to propose a solution to remove the strong reference mentioned above so that
GC can release FC entries to avoid long GC pause or OOM. It needs to be optional because this
change is a tradeoff, trading more CPU cycles for low memory footage. User can make final
decision depending on their use cases.
> The prototype attached use a combination of directory name and segment name as key to
the WeakHashMap, replacing the SegmentCoreReaders. Without change, Solr doesn't release any
FC entries after a GC is manually triggered. With the change, FC entries are usually released
after GC.
> However, I am not sure if it's the best way to solve this problem. Any suggestions are
welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message