spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-32715) Broadcast block pieces may memory leak
Date Tue, 15 Sep 2020 01:28:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dongjoon Hyun resolved SPARK-32715.
-----------------------------------
    Fix Version/s: 2.4.8
                   3.0.2
                   3.1.0
       Resolution: Fixed

Issue resolved by pull request 29558
[https://github.com/apache/spark/pull/29558]

> Broadcast block pieces may memory leak
> --------------------------------------
>
>                 Key: SPARK-32715
>                 URL: https://issues.apache.org/jira/browse/SPARK-32715
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.4.6, 3.0.0, 3.1.0
>            Reporter: Lantao Jin
>            Assignee: Lantao Jin
>            Priority: Major
>             Fix For: 3.1.0, 3.0.2, 2.4.8
>
>
> We use Spark thrift-server as a long-running service. A bad query submitted a heavy BroadcastNestLoopJoin
operation and made driver full GC. We killed the bad query but we found the driver's memory
usage was still high and full GCs had very frequency. By investigating with GC dump and log,
we found the broadcast may memory leak.
> 2020-08-19T18:54:02.824-0700: [Full GC (Allocation Failure) 2020-08-19T18:54:02.824-0700:
[Class Histogram (before full gc):
> 116G->112G(170G), 184.9121920 secs]
> [Eden: 32.0M(7616.0M)->0.0B(8704.0M) Survivors: 1088.0M->0.0B Heap: 116.4G(170.0G)->112.9G(170.0G)],
[Metaspace: 177285K->177270K(182272K)]
> num #instances #bytes class name
> ----------------------------------------------
> 1: 676531691 72035438432 [B
> 2: 676502528 32472121344 org.apache.spark.sql.catalyst.expressions.UnsafeRow
> 3: 99551 12018117568 [Ljava.lang.Object;
> 4: 26570 4349629040 [I
> 5: 6 3264536688 [Lorg.apache.spark.sql.catalyst.InternalRow;
> 6: 1708819 256299456 [C
> 7: 2338 179615208 [J
> 8: 1703669 54517408 java.lang.String
> 9: 103860 34896960 org.apache.spark.status.TaskDataWrapper
> 10: 177396 25545024 java.net.URI
> ...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message