hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11502) Map side aggregation is extremely slow
Date Mon, 10 Aug 2015 20:48:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680740#comment-14680740
] 

Gopal V commented on HIVE-11502:
--------------------------------

A custom hashcode can be used internal to Hive (i.e group-by etc), but not externally to hive
(bucketing into HDFS, results of hash() functions).

Because that would break external assumptions in a non-backwards-compatible way.

The reason shuffle + merge is more uniform is because it starts using [murmur hashes|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java#L366]
for UNIFORM trait RS instead of the builtin writable hash funcs (which are skewed).

You will probably notice that using a vectorized input format like ORC would not have the
issue you're hitting, since the vector transform inside the operator pipeline gives hive the
opportunity to use per-operator specific optimizations.

> Map side aggregation is extremely slow
> --------------------------------------
>
>                 Key: HIVE-11502
>                 URL: https://issues.apache.org/jira/browse/HIVE-11502
>             Project: Hive
>          Issue Type: Bug
>          Components: Logical Optimizer, Physical Optimizer
>    Affects Versions: 1.2.0
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
>
> For the query as following:
> {noformat}
> create table tbl2 as 
> select col1, max(col2) as col2 
> from tbl1 group by col1;
> {noformat}
> If the column for group by has many different values (for example 400000) and it is in
type double, the map side aggregation is very slow. I ran the query which took more than 3
hours , after 3 hours, I have to kill the query.
> The same query can finish in 7 seconds, if I turn off map side aggregation by:
> {noformat}
> set hive.map.aggr = false;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message