hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10631) create_table_core method has invalid update for Fast Stats
Date Thu, 13 Aug 2015 23:36:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696166#comment-14696166
] 

Hive QA commented on HIVE-10631:
--------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12750344/HIVE-10631.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9357 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4955/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4955/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4955/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12750344 - PreCommit-HIVE-TRUNK-Build

> create_table_core method has invalid update for Fast Stats
> ----------------------------------------------------------
>
>                 Key: HIVE-10631
>                 URL: https://issues.apache.org/jira/browse/HIVE-10631
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 1.0.0
>            Reporter: Dongwook Kwon
>            Assignee: Aaron Tokhy
>            Priority: Minor
>         Attachments: HIVE-10631-branch-1.0.patch, HIVE-10631.patch
>
>
> HiveMetaStore.create_table_core method calls MetaStoreUtils.updateUnpartitionedTableStatsFast
when hive.stats.autogather is on, however for partitioned table, this updateUnpartitionedTableStatsFast
call scanning warehouse dir and doesn't seem to use it. 
> "Fast Stats" was implemented by HIVE-3959
> https://github.com/apache/hive/blob/branch-1.0/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1363
> From create_table_core method
> {code}
>         if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVESTATSAUTOGATHER) &&
>             !MetaStoreUtils.isView(tbl)) {
>           if (tbl.getPartitionKeysSize() == 0)  { // Unpartitioned table
>             MetaStoreUtils.updateUnpartitionedTableStatsFast(db, tbl, wh, madeDir);
>           } else { // Partitioned table with no partitions.
>             MetaStoreUtils.updateUnpartitionedTableStatsFast(db, tbl, wh, true);
>           }
>         }
> {code}
> Particularly Line 1363: // Partitioned table with no partitions.
> {code}
> MetaStoreUtils.updateUnpartitionedTableStatsFast(db, tbl, wh, true);
> {code}
> This call ends up calling Warehouse.getFileStatusesForUnpartitionedTable and do nothing
in MetaStoreUtils.updateUnpartitionedTableStatsFast method due to newDir flag is always true
> Impact of this bug is minor with HDFS warehouse location(hive.metastore.warehouse.dir),
it could be big with S3 warehouse location especially for large existing partitions.
> Also the impact is heighten with HIVE-6727 when warehouse location is S3, basically it
could scan wrong S3 directory recursively and do nothing with it. I will add more detail of
cases in comments



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message