hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 范宜臻 (Jira) <j...@apache.org>
Subject [jira] [Updated] (HIVE-24080) CBO failed: Array and Map are not support when convert calcite types to hive types
Date Fri, 28 Aug 2020 02:53:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-24080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

范宜臻 updated HIVE-24080:
-----------------------
    Description: 
{code:java}
drop table temp.test_cbo_right;
CREATE EXTERNAL TABLE temp.test_cbo_right(
  `rid` string,
  `dtm` string,
  `map` map<string, string>,
  `tags` array <string>
);drop table temp.test_cbo_left;
CREATE EXTERNAL TABLE temp.test_cbo_left(
  `lid` string,
  `dtm` string
)

explain select
b.tags
from temp.test_cbo_left a
left join temp.test_cbo_right b on a.lid = b.rid
where b.dtm = '20200814'
and a.dtm = '20200813'{code}
 

The Problem will occur in  left join and select array type or map type column in right table(reverse
for right join or both for full join).

code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from calcite and then
convert it back to `optimizedAST`, the projection of b.tags has been optimized to be  RexCall
=> CAST(b.tags) in calcite code. Normally the projection should be an RexInputRef, but
in this case, calcite need cast `not null` to `nullable`.  However, TypeConverter in ASTConverter
has not support ARRAY and MAP.

 

NPE threw here:

org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor visitCall
{code:java}
HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
if (ht.args != null) {
  for (String castArg : ht.args)
    astBldr.add(HiveParser.Identifier, castArg);
}
{code}
The final result is cbo failed and skip CalcitePlan and cause partition pruning failed.

when table is large, sql will explode.

 

 I wonder which project should fix this problem

  was:
{code:java}
drop table temp.test_cbo_fyz_right;
CREATE EXTERNAL TABLE temp.test_cbo_right(
  `rid` string,
  `dtm` string,
  `map` map<string, string>,
  `tags` array <string>
);drop table temp.test_cbo_fyz_left;
CREATE EXTERNAL TABLE temp.test_cbo_left(
  `lid` string,
  `dtm` string
)

explain select
b.tags
from temp.test_cbo_left a
left join temp.test_cbo_right b on a.lid = b.rid
where b.dtm = '20200814'
and a.dtm = '20200813'{code}
 

The Problem will occur in  left join and select array type or map type column in right table(reverse
for right join or both for full join).

code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from calcite and then
convert it back to `optimizedAST`, the projection of b.tags has been optimized to be  RexCall
=> CAST(b.tags) in calcite code. Normally the projection should be an RexInputRef, but
in this case, calcite need cast `not null` to `nullable`.  However, TypeConverter in ASTConverter
has not support ARRAY and MAP.

 

NPE threw here:

org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor visitCall
{code:java}
HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
if (ht.args != null) {
  for (String castArg : ht.args)
    astBldr.add(HiveParser.Identifier, castArg);
}
{code}
The final result is cbo failed and skip CalcitePlan and cause partition pruning failed.

when table is large, sql will explode.

 

 I wonder which project should fix this problem


> CBO failed: Array and Map are not support when convert calcite types to hive types
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-24080
>                 URL: https://issues.apache.org/jira/browse/HIVE-24080
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO, Logical Optimizer
>    Affects Versions: 3.0.0, 2.3.4
>            Reporter: 范宜臻
>            Assignee: Ashutosh Chauhan
>            Priority: Blocker
>
> {code:java}
> drop table temp.test_cbo_right;
> CREATE EXTERNAL TABLE temp.test_cbo_right(
>   `rid` string,
>   `dtm` string,
>   `map` map<string, string>,
>   `tags` array <string>
> );drop table temp.test_cbo_left;
> CREATE EXTERNAL TABLE temp.test_cbo_left(
>   `lid` string,
>   `dtm` string
> )
> explain select
> b.tags
> from temp.test_cbo_left a
> left join temp.test_cbo_right b on a.lid = b.rid
> where b.dtm = '20200814'
> and a.dtm = '20200813'{code}
>  
> The Problem will occur in  left join and select array type or map type column in right
table(reverse for right join or both for full join).
> code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from calcite
and then convert it back to `optimizedAST`, the projection of b.tags has been optimized to
be  RexCall => CAST(b.tags) in calcite code. Normally the projection should be an RexInputRef,
but in this case, calcite need cast `not null` to `nullable`.  However, TypeConverter in
ASTConverter has not support ARRAY and MAP.
>  
> NPE threw here:
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor visitCall
> {code:java}
> HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
> ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
> if (ht.args != null) {
>   for (String castArg : ht.args)
>     astBldr.add(HiveParser.Identifier, castArg);
> }
> {code}
> The final result is cbo failed and skip CalcitePlan and cause partition pruning failed.
> when table is large, sql will explode.
>  
>  I wonder which project should fix this problem



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message