hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganesha Shreedhara (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Deleted] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
Date Mon, 02 Jul 2018 05:06:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ganesha Shreedhara updated HIVE-19850:
--------------------------------------
    Comment: was deleted

(was: Found that the value for TableScanOperator is not getting retrieved from 

rootToWorkMap even though operatorId/name/identifier is matching. It is not a good idea to
override equals/hashcode in Operator class as it can break other things, so added one more
level of comparing the table scan operators based on operatorId if the objects comparison
doesn't work. Please review and let me know if there is a better way of handling this.)

> Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-19850
>                 URL: https://issues.apache.org/jira/browse/HIVE-19850
>             Project: Hive
>          Issue Type: Bug
>          Components: Tez
>    Affects Versions: 3.0.0
>            Reporter: Ganesha Shreedhara
>            Assignee: Ganesha Shreedhara
>            Priority: Major
>         Attachments: HIVE-19850.patch
>
>
>  
> When multiple views are used along with union all, it is resulting in the following error
when dynamic partition pruning is enabled in tez. 
>  
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for tablescan TS[8]
>  at org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
>  at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
>  at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
>  at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
>  at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
>  at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
>  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
>  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
>  at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
>  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
>  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
>  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
>  at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
>  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
>  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>  
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean boolean)
partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean boolean)
partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean boolean)
partitioned by (dt string);
>  
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>  
> CREATE VIEW `view1` AS select `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
from `t1` union all select `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
from `t2`;
> CREATE VIEW `view2` AS select `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
from `t2` union all select `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union all select
v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1 v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
from `t4` union all select `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
from `t1`;
>  
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work found
for tablescan error



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message