lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaolu Zhao <xiaolu.z...@oracle.com>
Subject Re: Difference between unloading of cores with LotsOfCores and unloading a core with CoreAdmin
Date Wed, 22 Oct 2014 16:33:31 GMT
Hi Erick,

Thanks a lot for your explanation.

Last time, when I try out LotsOfCores, I find JVM memory usage will 
increase as the total number of cores grows, though the transient cache 
size is fixed. Finally, JVM will run out of memory when I have thousands 
of cores. Does it mean other currently unloaded cores will consume 
memory? Or swapping among loaded/unloaded cores will consume memory?

Best,
Xiaolu
On 10/22/2014 12:23 PM, Erick Erickson wrote:
> The difference here is that the LotsOfCores is intended to cache open
> cores and thus limit the number of currently loaded cores. However,
> cores not currently loaded are available for use; the next request
> that needs that core will cause it to be loaded (or reloaded).
>
> The admin/core/UNLOAD command, on the other hand, is designed to
> _permanently_ remove the core from Solr. Or at least have it become
> unavailable until another explicit admin/core command is executed to
> bring it back. There is nothing automatic about this.
>
> Another way of looking at it is that LotsOfCores is used in a
> situation where you don't know what requests are coming in, but you
> _can_ predict that not many will be used at once. So if I have 500
> cores, and my expectation is that only 20 of them are used at once,
> there's no good in having the 480 other cores loaded all the time.
> When a query comes in for one of the currently-unloaded cores (call it
> core21), that core is loaded (perhaps displacing one of the
> currently-loaded cores) and the request is served.
>
> If core21 above had been unloaded with the core/admin command, then a
> request directed to it would return an error instead.
>
> Best,
> Erick
>
> On Wed, Oct 22, 2014 at 12:11 PM, Xiaolu Zhao <xiaolu.zhao@oracle.com> wrote:
>> Hi All,
>>
>> I am confused about the difference between unloading of cores with
>> LotsOfCores and unloading a core with CoreAdmin.
>>
>>  From my understanding of LotsOfCores, if one core is removed from transient
>> cache, it is pending to close, it means close all resources allocated by the
>> core if it is no longer in use, e.g. searcher, updateHandler... While for
>> unloading a core with CoreAdmin, this core needs to be removed from the
>> cores list, either ordinary cores list or transient cores list, and cores
>> locator will delete it. If this core is loaded but not pending to close, it
>> will be close.
>>
>> Also, one more interesting thing is if I unload a core with CoreAdmin,
>> "core.properties" will be renamed "core.properties.unloaded". Then this core
>> cannot be found in the Solr API, and STATUS url won't return its status as
>> well. But with LotsOfCores, a core not in the transient cache will still
>> have "core.properties" and could be found through STATUS url, though it is
>> marked with "isLoaded=false".
>>
>> Could anyone tell me the underlying mechanism for these two cases? Why
>> LotsOfCores could realize frequent unloading/loading of cores? Do cores not
>> in the transient cores still consume JVM memory, while unloaded cores with
>> CoreAdmin not?
>>
>> Thanks,
>> Xiaolu


Mime
View raw message