spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jungtaek Lim (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-29301) Removing block is not reflected to the driver/executor's storage memory
Date Tue, 01 Oct 2019 02:27:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-29301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941473#comment-16941473
] 

Jungtaek Lim commented on SPARK-29301:
--------------------------------------

I'll push the branch-2.4 version of code in my repo and comment. Btw, there's some discussion
on how to deal with this issue (and other issue) so the patch may not be shipped as it is.
Please refer https://github.com/apache/spark/pull/25779

> Removing block is not reflected to the driver/executor's storage memory
> -----------------------------------------------------------------------
>
>                 Key: SPARK-29301
>                 URL: https://issues.apache.org/jira/browse/SPARK-29301
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.4.4, 3.0.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> While investigating SPARK-29055 I've found that heap memory in driver doesn't increase
which would denote memory leak happen, but storage memory doesn't decrease even there're some
broadcast blocks getting removed - these values keep increasing.
> Please refer SPARK-29055 to see the step to reproduce. Very easy to see it, as running
simple query repeatedly would reproduce the issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message