lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Muir <rcm...@gmail.com>
Subject Re: How to regulate native memory?
Date Thu, 31 Aug 2017 03:04:22 GMT
Hello,

>From the thread linked there, its not clear to me the problem relates
to lucene (vs being e.g. a bug in netty, or too many threads, or
potentially many other problems).

Can you first try to determine to breakdown your problematic "RSS"
from the operating system? Maybe this helps determine if your issue is
with an anonymous mapping (ByteBuffer.allocateDirect) or file mapping
(FileChannel.map).

WIth recent kernels you can break down RSS with /proc/pid/XXX/status
(RssAnon vs RssFile vs RssShmem):

    http://man7.org/linux/man-pages/man5/proc.5.html

If your kernel is old you may have to go through more trouble (summing
up stuff from smaps or whatever)

On Wed, Aug 30, 2017 at 9:58 PM, Erik Stephens <mrerikstephens@gmail.com> wrote:
> Our elasticsearch processes have been slowly consuming memory until a kernel OOM kills
it.  Details are here:
>
>     https://github.com/elastic/elasticsearch/issues/26269 <https://github.com/elastic/elasticsearch/issues/26269>
>
> To summarize:
>
> - Explicit GC is enabled
> - MaxDirectMemorySize is set
> - Total memory usage for the process is roughly heap (30G) + mmap'd (unbounded) + 1-2G
(meta space, threads, etc)
>
> The crowd is suggesting "Don't worry. You want to use all that memory."  I understand
that sentiment except for:
>
> - The process eventually gets OOM killed.
> - I need to support multiple processes on same machine and need a more predictable footprint.
>
> It seems to be relatively common knowledge that direct byte buffers require a GC to trigger
their freedom.  However, full GC's are happening but not resulting in a reduction of resident
mmap'd memory.
>
> Any pointers to source code, settings, or tools are much appreciated.  Thanks!
>
> --
> Erik
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message