hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21697) Remove periodical full refresh in HMS cache
Date Sat, 18 May 2019 01:23:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842840#comment-16842840
] 

Hive QA commented on HIVE-21697:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12969053/HIVE-21697.5.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 16052 tests executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.cli.TestHiveCli.testCmd (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testCommentStripping (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testDatabaseOptions (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testErrOutput (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testHelp (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testInValidCmd (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidDatabaseOptions (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions2 (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testSetHeaderValue (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testSetPromptValue (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd2 (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd3 (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd4 (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmd (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmdWithDBName (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB1 (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB2 (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB3 (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testUseInvalidDB (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testVariables (batchId=203)
org.apache.hive.beeline.cli.TestHiveCli.testVariablesForSource (batchId=203)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17246/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17246/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17246/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12969053 - PreCommit-HIVE-Build

> Remove periodical full refresh in HMS cache
> -------------------------------------------
>
>                 Key: HIVE-21697
>                 URL: https://issues.apache.org/jira/browse/HIVE-21697
>             Project: Hive
>          Issue Type: Improvement
>          Components: Standalone Metastore
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>            Priority: Major
>         Attachments: HIVE-21697.1.patch, HIVE-21697.2.patch, HIVE-21697.3.patch, HIVE-21697.4.patch,
HIVE-21697.5.patch
>
>
> In HIVE-18661, we added periodical notification based refresh in HMS cache. We shall
remove periodical full refresh to simplify the code as it will no longer be used. In the mean
time, we introduced mechanism to provide monotonic reads through the CachedStore.commitTransaction.
This will no longer be needed after HIVE-21637. So I will remove related code as well. This
will provide some performance benefits include:
> 1. We don't have to slow down write to catch up notification logs. Write can be done
immediately and tag the cache with writeids
> 2. We can read from cache even if updateUsingNotificationEvents is running. Read will
compare the writeids of the cache so monotonic reads will be guaranteed
> I'd like to put a patch separately with HIVE-21637 so it can be tested independently.
HMW will use periodical notification based refresh to update cache. And it will temporarily
lift the monotonic reads guarantee until HIVE-21637 checkin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message