lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gus Heck (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores
Date Sun, 21 Feb 2016 23:24:18 GMT

    [ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156290#comment-15156290
] 

Gus Heck commented on SOLR-8349:
--------------------------------

That sounds like a very cool way to store the source data, but this issue isn't about access
to the blob/file itself. It's about access to the in memory java object created from the file/blob/whatever,
so each core does not have to hold it's own identical copy in RAM. In our case we had a csv
file that was translated into a HashMap (for fast access *during* query processing) that had
>900MB of memory usage. When we fired up 40 cores (20 collections each with a main core
and one replica) that used this component RAM usage was through the roof with no data in the
index yet. 

A similar situation is faced by Analyzers that have dictionaries etc (see SOLR-3443). The
current patch no longer supports a solution to that issue, as support for Lucene layer stuff
has been deferred per discussion above.

It is interesting to see that there is an implementation of the ref counting to use as a model,
and your notes about hooks on the component will be useful too. That may come in handy if/when
a refresh mechanism is desired.

> Allow sharing of large in memory data structures across cores
> -------------------------------------------------------------
>
>                 Key: SOLR-8349
>                 URL: https://issues.apache.org/jira/browse/SOLR-8349
>             Project: Solr
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 5.3
>            Reporter: Gus Heck
>         Attachments: SOLR-8349.patch, SOLR-8349.patch
>
>
> In some cases search components or analysis classes may utilize a large dictionary or
other in-memory structure. When multiple cores are loaded with identical configurations utilizing
this large in memory structure, each core holds it's own copy in memory. This has been noted
in the past and a specific case reported in SOLR-3443. This patch provides a generalized capability,
and if accepted, this capability will then be used to fix SOLR-3443.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message