hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongzhi Chen (JIRA)" <>
Subject [jira] [Commented] (HIVE-11502) Map side aggregation is extremely slow
Date Wed, 12 Aug 2015 14:21:46 GMT


Yongzhi Chen commented on HIVE-11502:

The basic idea of the fix is in LazyDouble, I separate hashcode for internal use from hashcode
for hdfs.
The aggregation hashmap use LazyDouble, and I think hadoop use the value by DoubleWritable
object which is an instance variable in
LazyDouble (data). The change let LazyDouble calculate its own hashcode instead of blindly
use data's. 
I do not see risk here. You can assume the LazyDouble as a vectorized object which only has
one element in it.
Please correct me, if you find anything is not right. 

> Map side aggregation is extremely slow
> --------------------------------------
>                 Key: HIVE-11502
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Logical Optimizer, Physical Optimizer
>    Affects Versions: 1.2.0
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
>         Attachments: HIVE-11502.1.patch
> For the query as following:
> {noformat}
> create table tbl2 as 
> select col1, max(col2) as col2 
> from tbl1 group by col1;
> {noformat}
> If the column for group by has many different values (for example 400000) and it is in
type double, the map side aggregation is very slow. I ran the query which took more than 3
hours , after 3 hours, I have to kill the query.
> The same query can finish in 7 seconds, if I turn off map side aggregation by:
> {noformat}
> set = false;
> {noformat}

This message was sent by Atlassian JIRA

View raw message