mxnet-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ayres (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MXNET-1154) Update Module.Scala to resolve Memory Leaks
Date Tue, 23 Apr 2019 22:16:00 GMT

    [ https://issues.apache.org/jira/browse/MXNET-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824593#comment-16824593
] 

Andrew Ayres commented on MXNET-1154:
-------------------------------------

Documenting where this issues stands for when it's eventually picked back up.

Initially when we attempted to update module with ResourceScope, we ran into some problems
with the update causing segmentation faults. Some time was spent to try and resolve this but
little progress was made. Since this task is primarily about code maintenance and maintainability
(i.e. no known bugs, leaks, or other issues resulted from this), other tasks were given priority.
At the time, it was believed that the segmentation fault was the result of an incorrect implementation
and not due to any issues with ResourceScope itself.

Later, while fixing [https://github.com/apache/incubator-mxnet/issues/10867], it was realized
that the segmentation faults were occurring due to an unforeseen complication with ResourceScope.
Essentially, a native resource would be created, then new ResourceScopes opened, and the
native resource could end up creating new dependencies that were also native resources. The
result was that the original native resource was at one scope level and it's dependencies
at another. Leading to the resources dependencies being deallocated and leaving it in a bad
state.

The PR which fixed the bug likely fixed the cause of the seg faults that we were seeing as
well. At the very least, it introduced the mechanisms to resolve them (ResourceScope.usingIfScopeExists).

The original attempt to fix this is preserved at [https://github.com/andrewfayres/incubator-mxnet/tree/moodule].
This can mostly be copied but there will be a few sections which should be updated for the
changes to the code base since then. It may also be necessary to add the usingIfScopeExists
method inside module wherever native resources are created.

> Update Module.Scala to resolve Memory Leaks
> -------------------------------------------
>
>                 Key: MXNET-1154
>                 URL: https://issues.apache.org/jira/browse/MXNET-1154
>             Project: Apache MXNet
>          Issue Type: Sub-task
>          Components: Apache MXNet Scala API
>            Reporter: Naveen Swamy
>            Priority: Major
>
> Module.Scala throws a quite a few warnings for undisposed NDArrays. Although they will
be cleaned up at some point due to the Phantom Reference feature, it could be cause OOM if
the GC does not run frequently and also it is bad experience to see a lot of Warnings with
Stack traces. 
> Reach out to me offline and I will discuss some of the pointers I have found and possible
solutions.
> This is one of the issues that a Clojure user reported, which is applicable to both Scala
and Java users.
> https://discuss.mxnet.io/t/dealing-with-memory-leaks-ndarray/1767
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org


Mime
View raw message