lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: OOM with 2.9
Date Mon, 13 Jul 2009 09:52:31 GMT
Ahh good point.  I agree it makes sense to make MMapDir's chunking
user-controllable.  Can you open an issue?


On Mon, Jul 13, 2009 at 5:49 AM, Uwe Schindler<> wrote:
>> On Sun, Jul 12, 2009 at 10:51 AM, eks dev<> wrote:
>> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow...
>> maybe for such cases this threshold can become user settable. I will try
>> to experiment with it  (I am talking about MMapDirectory -> private final
>> int MAX_BBUF = Integer.MAX_VALUE;)
>> This chunking doesn't reduce virtual memory usage, because the buffers
>> are all mapped up front on opening the IndexInput.
>> It's only done because of a limitation of ByteBuffer (it uses int not
>> long to address the values).
> The idea of chunking is because of address space fragmentation. If the Java
> VM allocates or frees memory (e.g. when expanding the heap from Xms to Xmx),
> the address space of the process gets fragmented. If the index is very
> large, the byte buffer must fit in complete as one fragment into the address
> space, which may easily fail when you only have 32 bit.
> If the file is mapped in different chunks, each chunk can be somewhere else
> in address space (e.g., not only one 1 Gig junk is needed, there can be ten
> 100 Meg junks at different places).
> Uwe
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message