flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maximilian Michels (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4094) Off heap memory deallocation might not properly work
Date Tue, 02 Aug 2016 13:33:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403961#comment-15403961

Maximilian Michels commented on FLINK-4094:

Thanks [~StephanEwen] for clyariying that we can't release the memory ourselves. It seems
like the only two options we have is to

1) Pool off-heap memory regardless of the preallocation mode. That would change the semantics
of preallocation because it is supposed to eventually return memory.
2) Set the DirectMaxMemorySize limit correctly. This has been hard to do in the past because
calculatating how much memory Netty uses, is not so easy to do. Also we have two versions
of Netty, one included with Akka, one for the Flink network stack.

Both options seem not really viable. I think before we change memory allocation behavior,
we should discuss that on the Flink mailing list. 

[~ram_krish] What we can do now, is to discourage the use of off-heap memory with preallocation
set to false. For example, print a prominent warning and add a hint to the documentation.

> Off heap memory deallocation might not properly work
> ----------------------------------------------------
>                 Key: FLINK-4094
>                 URL: https://issues.apache.org/jira/browse/FLINK-4094
>             Project: Flink
>          Issue Type: Bug
>          Components: Local Runtime
>    Affects Versions: 1.1.0
>            Reporter: Till Rohrmann
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Critical
>             Fix For: 1.1.0
> A user reported that off-heap memory is not properly deallocated when setting {{taskmanager.memory.preallocate:false}}
(per default) [1]. This can cause the TaskManager process being killed by the OS.
> It should be possible to execute multiple batch jobs with preallocation turned off. No
longer used direct memory buffers should be properly garbage collected so that the JVM process
does not exceed it's maximum memory bounds.
> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/offheap-memory-allocation-and-memory-leak-bug-td12154.html

This message was sent by Atlassian JIRA

View raw message