spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Masiero Vanzin (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-26418) Only OpenBlocks without any ChunkFetch for one stream will cause memory leak in ExternalShuffleService
Date Fri, 20 Dec 2019 19:33:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Masiero Vanzin resolved SPARK-26418.
--------------------------------------------
    Resolution: Duplicate

> Only OpenBlocks without any ChunkFetch for one stream will cause memory leak in ExternalShuffleService
> ------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-26418
>                 URL: https://issues.apache.org/jira/browse/SPARK-26418
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>    Affects Versions: 2.4.0
>            Reporter: Wang Shuo
>            Priority: Major
>
> In current code path,  OneForOneStreamManager holds StreamState in a Map named streams. 
> A StreamState is initialized and put into streams when OpenBlocks request received.
> One specific StreamState is removed from streams in two scenarios below:
>  # The last chunk of a stream is fetched
>  # The connection of ChunkFetch is closed
> StreamState will never be clean up, if OpenBlocks request is received without and following 
ChunkFetch request. This will cause memory leak in server side, which is harmful for long
running service such as ExternalShuffleService.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message