hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
Date Sun, 14 Jul 2019 10:26:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884632#comment-16884632
] 

Hive QA commented on HIVE-21225:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12974627/HIVE-21225.10.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 16627 tests executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=177)
	[union_top_level.q,vector_create_struct_table.q,llap_uncompressed.q,schema_evol_text_vecrow_part_all_primitive.q,materialized_view_partitioned_2.q,murmur_hash_migration.q,update_where_partitioned.q,materialized_view_rewrite_1.q,materialized_view_create_rewrite_time_window_2.q,drop_partition_with_stats.q,vectorization_1.q,skiphf_aggr.q,fold_varchar.q,auto_join_filters.q,join0.q,insert_orig_table.q,mergejoin.q,vector_if_expr_2.q,orc_split_elimination.q,vector_outer_join0.q,vector_udf1.q,schema_evol_text_vec_part_all_primitive.q,vector_complex_all.q,union3.q,windowing_windowspec2.q,auto_smb_mapjoin_14.q,vector_mapjoin_complex_values.q,results_cache_truncate.q,vector_join_filters.q,reduce_deduplicate_extended.q]
org.apache.hadoop.hive.ql.TestAcidOnTez.testBucketedAcidInsertWithRemoveUnion (batchId=244)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCrudMajorCompactionSplitGrouper (batchId=244)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=330)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=332)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerialize (batchId=321)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas
(batchId=321)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerialize (batchId=321)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas
(batchId=321)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testFSCallsVectorizedOrcAcidRowBatchReader
(batchId=321)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps (batchId=321)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache (batchId=321)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeFileLen
(batchId=321)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeModificationTime
(batchId=321)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTblNew (batchId=246)
org.apache.hadoop.hive.ql.util.TestUpgradeTool.testPostUpgrade (batchId=306)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18023/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18023/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18023/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12974627 - PreCommit-HIVE-Build

> ACID: getAcidState() should cache a recursive dir listing locally
> -----------------------------------------------------------------
>
>                 Key: HIVE-21225
>                 URL: https://issues.apache.org/jira/browse/HIVE-21225
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>            Reporter: Gopal V
>            Assignee: Vaibhav Gumashta
>            Priority: Major
>         Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, HIVE-21225.2.patch, HIVE-21225.3.patch,
HIVE-21225.4.patch, HIVE-21225.4.patch, HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch,
HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg
>
>
> Currently getAcidState() makes 3 calls into the FS api which could be answered by making
a single recursive listDir call and reusing the same data to check for isRawFormat() and isValidBase().
> All delta operations for a single partition can go against a single listed directory
snapshot instead of interacting with the NameNode or ObjectStore within the inner loop.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message