hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20284) In strict mode, if constant propagation is enable, the partition filter is folded before partition pruner lead to error "No partition predicate for Alias"
Date Wed, 01 Aug 2018 06:02:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564782#comment-16564782
] 

Hive QA commented on HIVE-20284:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933861/HIVE-20284.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12980/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12980/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12980/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ date '+%Y-%m-%d %T.%3N'
2018-08-01 06:01:16.260
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12980/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-08-01 06:01:16.263
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 4d43695 HIVE-18201 : Disable XPROD_EDGE for sq_count_check()  created for scalar
subqueries (Ashutosh Chauhan via Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 4d43695 HIVE-18201 : Disable XPROD_EDGE for sq_count_check()  created for scalar
subqueries (Ashutosh Chauhan via Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-08-01 06:01:17.302
+ rm -rf ../yetus_PreCommit-HIVE-Build-12980
+ mkdir ../yetus_PreCommit-HIVE-Build-12980
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12980
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12980/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
error: patch failed: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:4535
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' with conflicts.
error: patch failed: ql/src/test/results/clientnegative/alter_view_failure6.q.out:19
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientnegative/alter_view_failure6.q.out' with conflicts.
error: patch failed: ql/src/test/results/clientnegative/input_part0_neg.q.out:1
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientnegative/input_part0_neg.q.out' with conflicts.
error: patch failed: ql/src/test/results/clientnegative/strict_pruning.q.out:1
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientnegative/strict_pruning.q.out' with conflicts.
Going to apply patch with: git apply -p0
error: patch failed: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:4535
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' with conflicts.
error: patch failed: ql/src/test/results/clientnegative/alter_view_failure6.q.out:19
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientnegative/alter_view_failure6.q.out' with conflicts.
error: patch failed: ql/src/test/results/clientnegative/input_part0_neg.q.out:1
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientnegative/input_part0_neg.q.out' with conflicts.
error: patch failed: ql/src/test/results/clientnegative/strict_pruning.q.out:1
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientnegative/strict_pruning.q.out' with conflicts.
U common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
U ql/src/test/results/clientnegative/alter_view_failure6.q.out
U ql/src/test/results/clientnegative/input_part0_neg.q.out
U ql/src/test/results/clientnegative/strict_pruning.q.out
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12980
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12933861 - PreCommit-HIVE-Build

> In strict mode, if constant propagation is enable, the partition filter is folded before
partition pruner lead to error "No partition predicate for Alias"  
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-20284
>                 URL: https://issues.apache.org/jira/browse/HIVE-20284
>             Project: Hive
>          Issue Type: Improvement
>          Components: CLI
>    Affects Versions: 1.2.1, 2.3.3
>            Reporter: Hui Huang
>            Assignee: Hui Huang
>            Priority: Trivial
>             Fix For: 2.3.3
>
>         Attachments: HIVE-20284.patch
>
>
> In strict mode and the hive.optimize.constant.propagation is set to true, the following
sql will failed:
> {code:java}
> hive> desc employee_part;
> OK
> col_name	data_type	comment
> eid                 	int
> name                	string
> dept                	string
> year                	string
> month               	string
> # Partition Information
> # col_name            	data_type           	comment
> year                	string
> month               	string
> Time taken: 0.564 seconds, Fetched: 11 row(s)
> hive> set hive.mapred.mode=strict;
> hive> select * from employee_part where false and concat(year,month)='201807';
> FAILED: SemanticException Queries against partitioned tables without a partition filter
are disabled for safety reasons. If you know what you are doing, please sethive.strict.checks.large.query
to false and that hive.mapred.mode is not set to 'strict' to proceed. Note that if you may
get errors or incorrect results if you make a mistake while using some of the unsafe features.
No partition predicate for Alias "employee_part" Table "employee_part"
> {code}
> The above error msg is confusing,  concat(year,month)='201807' is the partition filter。
> The reason is during logic optimization, the ConstantPropagate optimizer is running before
partitionPruner optimizer, when found a express like 'false and concat(year,month)=xxxx',
the express is replaced with 'fasle' , and the partition filter is dropped. So the PartitionPruner
can not get the partition filter.
> Users can remove the constant express that always has true/false values to work around.
> When views used, if some columns are constant values, users  will be  confusing.
> So we should add some more message in the error msg returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message