spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-26525) Fast release memory of ShuffleBlockFetcherIterator
Date Mon, 11 Feb 2019 04:30:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenchen Fan resolved SPARK-26525.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.0

Issue resolved by pull request 23438
[https://github.com/apache/spark/pull/23438]

> Fast release memory of ShuffleBlockFetcherIterator
> --------------------------------------------------
>
>                 Key: SPARK-26525
>                 URL: https://issues.apache.org/jira/browse/SPARK-26525
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle
>    Affects Versions: 2.3.2
>            Reporter: liupengcheng
>            Assignee: liupengcheng
>            Priority: Major
>             Fix For: 3.0.0
>
>
> Currently, spark would not release ShuffleBlockFetcherIterator until the whole task finished.
> In some conditions, it incurs memory leak.
> An example is Shuffle -> map -> Coalesce(shuffle = false). Each ShuffleBlockFetcherIterator
contains  some metas about MapStatus(blocksByAddress) and each ShuffleMapTask will keep n(max
to shuffle partitions) shuffleBlockFetcherIterator for they are refered by onCompleteCallbacks
of TaskContext, in some case, it may take huge memory and the memory will not released until
the task finished.
> Actually, We can release ShuffleBlockFetcherIterator as soon as it's consumed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message