lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores
Date Sun, 21 Feb 2016 08:45:18 GMT

    [ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155958#comment-15155958
] 

David Smiley commented on SOLR-8349:
------------------------------------

I now see Guava {{Cache.get(key,Callable loader)}} (and you use it in the patch), so that
helps a lot to keep things simple.

In #3 what do you mean by "deterministic behavior"?  Do you mean, having code that implements
finalize() or uses a ReferenceQueue, or basically needs to be closed in some way?  Normally,
we don't care the exact means of when the VM literally frees memory, so I doubt you mean that.
 For objects that implement AutoCloseable, we might want to throw an exception to state we
don't support closing the object.  Support for that would be nice; maybe via putting hard
references to just those in the CoreContainer and having them be iterated and closed when
the CoreContainer is closed.

#4: I'd rather avoid new abstractions unless it's clearly useful, and so I question the utility
of the interfaces ContainerResourceAware & ContainerResourceProvider.  A component that
wants the cache could implement SolrCoreAware and call core.getCoreDescriptor().getCoreContainer()
which could expose one method like getOrLoadCached(key,lambdaLoader).  For components that
can't implement SolrCoreAware, it could implement ResourceLoaderAware and cast the loader
to SolrResourceLoader and we could expose access to the CoreContainer from there with a new
method.  Is there some sequencing issue that makes this impossible?

Furthermore, there's some complexity in your patch in trying to guarantee a hard reference,
but I don't see that as necessary.  If some component calls this cache method, it will then
immediately puts it some place like a field of the component (be it SearchComponent or TokenFilterFactory
or whatever).  Similar story for the hypothetical future issue to propose having the SolrCore
put entire SearchComponents and/or TokenFilters (etc.) into this cache.

Can we use Object keys please?  Remember Node.java example earlier.

> Allow sharing of large in memory data structures across cores
> -------------------------------------------------------------
>
>                 Key: SOLR-8349
>                 URL: https://issues.apache.org/jira/browse/SOLR-8349
>             Project: Solr
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 5.3
>            Reporter: Gus Heck
>         Attachments: SOLR-8349.patch, SOLR-8349.patch
>
>
> In some cases search components or analysis classes may utilize a large dictionary or
other in-memory structure. When multiple cores are loaded with identical configurations utilizing
this large in memory structure, each core holds it's own copy in memory. This has been noted
in the past and a specific case reported in SOLR-3443. This patch provides a generalized capability,
and if accepted, this capability will then be used to fix SOLR-3443.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message