hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongzhi Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11502) Map side aggregation is extremely slow
Date Tue, 11 Aug 2015 00:33:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681015#comment-14681015
] 

Yongzhi Chen commented on HIVE-11502:
-------------------------------------

[~gopalv], thanks for the workaround. But I am afraid some users do not want to change their
input format. And this HashMap may affect mapjoin too. We help a user workaround this map
side aggregation issue by set hive.map.aggr = false; After that, the simple group test case
has very good performance, but a more complicated join query with group by as subquery stuck
on mapjoin. So we have to let the user turn off mapjoin by set hive.auto.convert.join=false;
 The performance hit by this bug is really outstanding. Without workaround, none of the query
can finish in several hours. So I think we have to fix it. 

> Map side aggregation is extremely slow
> --------------------------------------
>
>                 Key: HIVE-11502
>                 URL: https://issues.apache.org/jira/browse/HIVE-11502
>             Project: Hive
>          Issue Type: Bug
>          Components: Logical Optimizer, Physical Optimizer
>    Affects Versions: 1.2.0
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
>
> For the query as following:
> {noformat}
> create table tbl2 as 
> select col1, max(col2) as col2 
> from tbl1 group by col1;
> {noformat}
> If the column for group by has many different values (for example 400000) and it is in
type double, the map side aggregation is very slow. I ran the query which took more than 3
hours , after 3 hours, I have to kill the query.
> The same query can finish in 7 seconds, if I turn off map side aggregation by:
> {noformat}
> set hive.map.aggr = false;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message