hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20432) Rewrite BETWEEN to IN for integer types for stats estimation
Date Tue, 21 Aug 2018 09:32:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587209#comment-16587209
] 

Hive QA commented on HIVE-20432:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936374/HIVE-20432.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 63 failed/errored test(s), 14885 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join46] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin46] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_46] (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_47] (batchId=31)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1] (batchId=194)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4] (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_predicate_pushdown]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_predicate_pushdown]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select] (batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_3]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select] (batchId=127)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query10] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query22] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query25] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query28] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query29] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query34] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query37] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query38] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query51] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query59] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query65] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query66] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query67] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query68] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query69] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query70] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query73] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query79] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query82] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query86] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query87] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query90] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query97] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query99] (batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query9] (batchId=266)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query10] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query22] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query25] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query28] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query29] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query34] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query37] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query38] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query51] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query59] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query65] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query66] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query67] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query68] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query69] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query70] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query73] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query79] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query82] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query86] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query87] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query90] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query97] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query99] (batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query9] (batchId=264)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13367/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13367/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13367/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 63 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12936374 - PreCommit-HIVE-Build

> Rewrite BETWEEN to IN for integer types for stats estimation
> ------------------------------------------------------------
>
>                 Key: HIVE-20432
>                 URL: https://issues.apache.org/jira/browse/HIVE-20432
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 4.0.0
>            Reporter: Vineet Garg
>            Assignee: Vineet Garg
>            Priority: Major
>         Attachments: HIVE-20432.1.patch
>
>
> IN clause provides better statistics estimation since predicates are equality based and
it keeps into account selectivity of all values. BETWEEN is rewritten into {{>= AND <=}}
i.e. non-equality predicates which cause worse estimation. For integer types BETWEEN could
be rewritten into IN e.g. BETWEEN 4 and 7 is equivalent to IN (4,5,6,7).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message