lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Call <brian.c...@soterawireless.com>
Subject Re: Caused by: java.lang.OutOfMemoryError: Map failed
Date Sat, 08 Nov 2014 00:39:49 GMT
Half of those indexes max out at about 1.3G, the other half will always stay very small <
5m total. We keep an index for “raw” data and another index for events and “trended”
data. Possible design changes may make this number go up to 4-5G per index, but definitely
no more than that. 

We’re indexing streaming patient vitals data from many different devices simultaneously,
hence the high number of necessary indices. There are also constraints on patient identity
confirmation, which requires that multiple indices are created to keep unconfirmed patient
data separate from confirmed data. 

Also, we’re not using Solr, only raw lucene. The indices remain open until the streaming
data has stopped and a user has removed the related session from the UI. 

Yes, it’s a necessary kind of scary… 

-Brian

On Nov 7, 2014, at 4:20 PM, Erick Erickson <erickerickson@gmail.com> wrote:

> bq: Our server runs many hundreds (soon to be thousands) of indexes
> simultaneously
> 
> This is actually kind of scary. How do you expect to fit "many
> thousands" of indexes into
> memory? Raising per-process virtual memory to unlimited still doesn't handle the
> amount of RAM the Solr process needs. It holds things like caches,
> (top-level and
> per-segment), sort lists, all that. How many G of indexes are we
> talking here? Note
> that this is not a great guide to RAM requirements, but I'm just
> trying to get a handle
> on the scale you're at. You're not, for instance, going to handle
> terabyte-scale indexes
> on a single machine satisfactorily IMO.
> 
> If your usage pattern is a user signs on, works with their index for a
> while then
> signs off, you might get some joy out of the LotsOfCores option. That said, this
> option has NOT been validated on cloud setups, where I expect it'll
> have problems.
> 
> FWIW,
> Erick
> 
> On Fri, Nov 7, 2014 at 2:24 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
>> Hi,
>> 
>>> That error can also be thrown when the number of open files exceeds the
>>> given limit. "OutOfMemory" should really have been named
>>> "OutOfResources".
>> 
>> This was changed already. Lucene no longer prints OOM (it removes the OOM from stack
trace). It also adds useful information. So I think the version of Lucene that produced this
exception is older (before 4.9): https://issues.apache.org/jira/browse/LUCENE-5673
>> 
>> Uwe
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message