hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <>
Subject [jira] [Commented] (HIVE-11502) Map side aggregation is extremely slow
Date Mon, 10 Aug 2015 20:48:45 GMT


Gopal V commented on HIVE-11502:

A custom hashcode can be used internal to Hive (i.e group-by etc), but not externally to hive
(bucketing into HDFS, results of hash() functions).

Because that would break external assumptions in a non-backwards-compatible way.

The reason shuffle + merge is more uniform is because it starts using [murmur hashes|]
for UNIFORM trait RS instead of the builtin writable hash funcs (which are skewed).

You will probably notice that using a vectorized input format like ORC would not have the
issue you're hitting, since the vector transform inside the operator pipeline gives hive the
opportunity to use per-operator specific optimizations.

> Map side aggregation is extremely slow
> --------------------------------------
>                 Key: HIVE-11502
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Logical Optimizer, Physical Optimizer
>    Affects Versions: 1.2.0
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
> For the query as following:
> {noformat}
> create table tbl2 as 
> select col1, max(col2) as col2 
> from tbl1 group by col1;
> {noformat}
> If the column for group by has many different values (for example 400000) and it is in
type double, the map side aggregation is very slow. I ran the query which took more than 3
hours , after 3 hours, I have to kill the query.
> The same query can finish in 7 seconds, if I turn off map side aggregation by:
> {noformat}
> set = false;
> {noformat}

This message was sent by Atlassian JIRA

View raw message