hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesus Camacho Rodriguez (JIRA)" <>
Subject [jira] [Commented] (HIVE-10455) CBO (Calcite Return Path): Different data types at Reducer before JoinOp
Date Fri, 24 Apr 2015 21:44:39 GMT


Jesus Camacho Rodriguez commented on HIVE-10455:

[~pxiong], another option would be to extend HiveInsertExchange4JoinRule, so the RexNodes
for the keys of the Join are stored in HiveSortExchange. Then, the translation in HiveOpConverter
would not need to carry information from the Join above, but rather the HiveSortExchange would
have all the information it needs. I think it would become a bit cleaner. What do you think?

> CBO (Calcite Return Path): Different data types at Reducer before JoinOp
> ------------------------------------------------------------------------
>                 Key: HIVE-10455
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: CBO
>            Reporter: Pengcheng Xiong
>            Assignee: Pengcheng Xiong
>             Fix For: 1.2.0
>         Attachments: HIVE-10455.01.patch
> The following error occured for cbo_subq_not_in.q 
> {code}
> java.lang.Exception: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error: Unable to deserialize reduce input key from x1x128x0x0x1 with properties
{columns=reducesinkkey0, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe,
serialization.sort.order=+, columns.types=double}
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(
>         at org.apache.hadoop.mapred.LocalJobRunner$
> {code}
> A more easier way to reproduce is 
> {code}
> set hive.cbo.enable=true;
> set hive.exec.check.crossproducts=false;
> set hive.stats.fetch.column.stats=true;
> set;
> select p_size, src.key
> from 
> part join src
> on p_size=key;
> {code}
> As you can see, p_size is integer while src.key is string. Both of them should be cast
to double when they join. When return path is off, this will happen before Join, at RS. However,
when return path is on, this will be considered as an expression in Join. Thus, when reducer
is collecting different types of keys from different join branches, it throws exception.

This message was sent by Atlassian JIRA

View raw message