hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11306) Add a bloom-1 filter for Hybrid MapJoin spills
Date Fri, 14 Aug 2015 08:02:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696671#comment-14696671
] 

Gopal V commented on HIVE-11306:
--------------------------------

The patch .3 does not give performance boost observed in patch .2.

The crucial difference is that patch .3 does not really consider the bloom filter to be valid
for spilled partitions.

{code}
+      if (!bloom1.testLong(keyHash) && !isOnDisk(partitionId)) {
{code}

the isOnDisk check negates all the performance benefits of checking the bloom filter to avoid
spilling.

> Add a bloom-1 filter for Hybrid MapJoin spills
> ----------------------------------------------
>
>                 Key: HIVE-11306
>                 URL: https://issues.apache.org/jira/browse/HIVE-11306
>             Project: Hive
>          Issue Type: Improvement
>          Components: Hive
>    Affects Versions: 1.3.0, 2.0.0
>            Reporter: Gopal V
>            Assignee: Gopal V
>         Attachments: HIVE-11306.1.patch, HIVE-11306.2.patch, HIVE-11306.3.patch
>
>
> HIVE-9277 implemented Spillable joins for Tez, which suffers from a corner-case performance
issue when joining wide small tables against a narrow big table (like a user info table join
events stream).
> The fact that the wide table is spilled causes extra IO, even though the nDV of the join
key might be in the thousands.
> A cheap bloom-1 filter would add a massive performance gain for such queries, massively
cutting down on the spill IO costs for the big-table spills.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message