lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores
Date Fri, 12 Feb 2016 04:28:18 GMT

    [ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144017#comment-15144017
] 

David Smiley commented on SOLR-8349:
------------------------------------

1. the classLoader separation is simply because there are libs and conf/ resources per-core,
as cores might be for entirely separate purposes with different dependencies that might not
even be compatible if it were all together.
2. RE memory-leak:  I propose the weakValues() feature of Guava's cache.  It's a nice cache
:-)  No reference counting needed; we're in a VM that GCs.
3. you're right; this isn't just about analysis components.  But it would probably be incompatible
with anything that implements CoreAware or SchemaAware as both those things are tied to a
SolrCore.
4. RE "what's left to be loaded in the core level loader": I think this re-use feature should
be explicitly opt-in.  Most stuff will cotinue as-is -- core level loader.  Docs for HunspellFactory
on the ref guide could be augmented to suggest flagging global-reuse because we know it uses
a lot of memory.
5. RE keys: I think just the W3C DOM Node.java instance is probably fine.  We might want to
double-check it's equals() method makes sense and doesn't refer to the parent, or if it does
find a way to clone/detach it.  Actually I think it does; so we'd call cloneNode(true) and
use the result of that.  I'm not concerned on the efficiency of the keys; this cache isn't
going to be hit hard.

Any way, lets continue this discussion if need be on a follow-on issue for actually _using_
the cache that this issue will create.  Cool?

> Allow sharing of large in memory data structures across cores
> -------------------------------------------------------------
>
>                 Key: SOLR-8349
>                 URL: https://issues.apache.org/jira/browse/SOLR-8349
>             Project: Solr
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 5.3
>            Reporter: Gus Heck
>         Attachments: SOLR-8349.patch
>
>
> In some cases search components or analysis classes may utilize a large dictionary or
other in-memory structure. When multiple cores are loaded with identical configurations utilizing
this large in memory structure, each core holds it's own copy in memory. This has been noted
in the past and a specific case reported in SOLR-3443. This patch provides a generalized capability,
and if accepted, this capability will then be used to fix SOLR-3443.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message