spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-26525) Fast release memory of ShuffleBlockFetcherIterator
Date Fri, 01 Feb 2019 05:49:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenchen Fan reassigned SPARK-26525:
-----------------------------------

    Assignee: liupengcheng

> Fast release memory of ShuffleBlockFetcherIterator
> --------------------------------------------------
>
>                 Key: SPARK-26525
>                 URL: https://issues.apache.org/jira/browse/SPARK-26525
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle
>    Affects Versions: 2.3.2
>            Reporter: liupengcheng
>            Assignee: liupengcheng
>            Priority: Major
>
> Currently, spark would not release ShuffleBlockFetcherIterator until the whole task finished.
> In some conditions, it incurs memory leak.
> An example is Shuffle -> map -> Coalesce(shuffle = false). Each ShuffleBlockFetcherIterator
contains  some metas about MapStatus(blocksByAddress) and each ShuffleMapTask will keep n(max
to shuffle partitions) shuffleBlockFetcherIterator for they are refered by onCompleteCallbacks
of TaskContext, in some case, it may take huge memory and the memory will not released until
the task finished.
> Actually, We can release ShuffleBlockFetcherIterator as soon as it's consumed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message