spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Li (JIRA)" <>
Subject [jira] [Updated] (SPARK-22170) Broadcast join holds an extra copy of rows in driver memory
Date Mon, 09 Oct 2017 22:24:00 GMT


Xiao Li updated SPARK-22170:
    Issue Type: Improvement  (was: Bug)

> Broadcast join holds an extra copy of rows in driver memory
> -----------------------------------------------------------
>                 Key: SPARK-22170
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.0.2, 2.1.1, 2.2.0
>            Reporter: Ryan Blue
>            Assignee: Ryan Blue
>             Fix For: 2.3.0
> I investigated a driver OOM that was building a large broadcast table with a memory profiler
and found that a huge amount of memory is used while building a broadcast table. This is because
[BroadcastExchangeExec uses {{executeCollect}}|].
In {{executeCollect}}, all of the partitions are fetched as compressed blocks, then each block
is decompressed (with a stream), and each row is copied to a new byte buffer and added to
an ArrayBuffer, which is copied to an Array. This results in a huge amount of allocation:
a buffer for each row in the broadcast. Those rows are only used to get copied into a {{BytesToBytesMap}}
that will be broadcasted, so there is no need to keep them in memory.
> Replacing the array buffer step with an iterator reduces the amount of memory held while
creating the map by not requiring all rows to be in memory. It also avoids allocating a large
Array for the rows. In practice, a 16MB broadcast table used 100MB less memory with this approach,
but the reduction depends on the size of rows and compression (16MB was in Parquet format).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message