hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 范宜臻 (Jira) <j...@apache.org>
Subject [jira] [Updated] (HIVE-24080) CBO failed: Array and Map are not support when convert calcite types to hive types
Date Thu, 27 Aug 2020 09:46:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-24080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

范宜臻 updated HIVE-24080:
-----------------------
    Priority: Blocker  (was: Critical)

> CBO failed: Array and Map are not support when convert calcite types to hive types
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-24080
>                 URL: https://issues.apache.org/jira/browse/HIVE-24080
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO, Logical Optimizer
>    Affects Versions: 3.0.0, 2.3.4
>            Reporter: 范宜臻
>            Assignee: Ashutosh Chauhan
>            Priority: Blocker
>
> {code:java}
> drop table temp.test_cbo_fyz_right;
> CREATE EXTERNAL TABLE temp.test_cbo_right(
>   `rid` string,
>   `dtm` string,
>   `map` map<string, string>,
>   `tags` array <string>
> );drop table temp.test_cbo_fyz_left;
> CREATE EXTERNAL TABLE temp.test_cbo_left(
>   `lid` string,
>   `dtm` string
> )
> explain select
> b.tags
> from temp.test_cbo_left a
> left join temp.test_cbo_right b on a.lid = b.rid
> where b.dtm = '20200814'
> and a.dtm = '20200813'{code}
>  
> The Problem will occur in  left join and select array type or map type column in right
table(reverse for right join or both for full join).
> code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from calcite
and then convert it back to `optimizedAST`, the projection of b.tags has been optimized to
be  RexCall => CAST(b.tags) in calcite code. Normally the projection should be an RexInputRef,
but in this case, calcite need cast `not null` to `nullable`.  However, TypeConverter in
ASTConverter has not support ARRAY and MAP.
>  
> NPE threw here:
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor visitCall
> {code:java}
> HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
> ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
> if (ht.args != null) {
>   for (String castArg : ht.args)
>     astBldr.add(HiveParser.Identifier, castArg);
> }
> {code}
> The final result is cbo failed and skip CalcitePlan and cause partition pruning failed.
> when table is large, sql will explode.
>  
>  I wonder which project should fix this problem



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message