lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-5714) You should be able to use one pool of memory for multiple collection's HDFS block caches.
Date Tue, 04 Mar 2014 01:20:20 GMT

    [ https://issues.apache.org/jira/browse/SOLR-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918881#comment-13918881
] 

Mark Miller commented on SOLR-5714:
-----------------------------------

bq. Are there any downsides to using a global block cache? In the motivation above, it's to
promote sharing on the same index, but we are sharing globally. Why not share on just the
same index?

There are not a lot of downsides. Perhaps some extra contention if you have a ton of collections
and perhaps different LRU behavior across collections than you might want for some use cases.
I think by and large, the benefits of one global pool of memory will out weigh those smaller
issues by a good extent. Trying to figure out how much to give each collection is fairly difficult
- and some collections may be starving for RAM at certain times while other collections are
using very little of their pre allocated RAM. Unless you have so much RAM that you can just
give gobs of it to each SolrCore, I imagine one pool will be much more user friendly and efficient
in the standard case.

> You should be able to use one pool of memory for multiple collection's HDFS block caches.
> -----------------------------------------------------------------------------------------
>
>                 Key: SOLR-5714
>                 URL: https://issues.apache.org/jira/browse/SOLR-5714
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Mark Miller
>            Assignee: Mark Miller
>             Fix For: 4.8, 5.0
>
>         Attachments: SOLR-5714.patch, SOLR-5714.patch, SOLR-5714.patch, SOLR-5714.patch
>
>
> Currently, you have to specify how much direct memory to allocate per SolrCore. This
can be inefficient, and has some negative consequences - for instance, when replicating, many
times two HDFS directories will exist for the same index briefly, which will double the RAM
used for that SolrCore.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message