hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21697) Remove periodical full refresh in HMS cache
Date Sat, 18 May 2019 00:51:01 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842834#comment-16842834
] 

Hive QA commented on HIVE-21697:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 12s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 28s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 55s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 33s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 43s{color} | {color:blue}
standalone-metastore/metastore-common in master has 31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 34s{color} | {color:blue}
common in master has 62 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m 12s{color} | {color:blue}
standalone-metastore/metastore-server in master has 183 extant Findbugs warnings. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 27s{color} | {color:blue}
hcatalog/server-extensions in master has 3 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 44s{color} | {color:blue}
itests/hive-unit in master has 2 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 50s{color} | {color:blue}
itests/util in master has 44 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 30s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 27s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 27s{color} | {color:red}
server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 44s{color} | {color:red}
hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 19s{color} | {color:red}
server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 45s{color} | {color:red}
hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 19s{color} | {color:red}
server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 45s{color} | {color:red}
hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 29s{color} | {color:red}
standalone-metastore/metastore-server: The patch generated 62 new + 615 unchanged - 10 fixed
= 677 total (was 625) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 18s{color} | {color:red}
itests/hive-unit: The patch generated 3 new + 16 unchanged - 0 fixed = 19 total (was 16) {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 24s{color} | {color:red}
standalone-metastore/metastore-server generated 2 new + 182 unchanged - 1 fixed = 184 total
(was 183) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 19s{color} | {color:red}
server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 40s{color} | {color:red}
hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 34s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 15s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 22s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
|  |  Load of known null value in org.apache.hadoop.hive.metastore.cache.CachedStore.addPartitions(String,
String, String, PartitionSpecProxy, boolean)  At CachedStore.java:in org.apache.hadoop.hive.metastore.cache.CachedStore.addPartitions(String,
String, String, PartitionSpecProxy, boolean)  At CachedStore.java:[line 1065] |
|  |  Load of known null value in org.apache.hadoop.hive.metastore.cache.CachedStore.get_aggr_stats_for(String,
String, String, List, List, String)  At CachedStore.java:in org.apache.hadoop.hive.metastore.cache.CachedStore.get_aggr_stats_for(String,
String, String, List, List, String)  At CachedStore.java:[line 2039] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17246/dev-support/hive-personality.sh
|
| git revision | master / 9a10bc2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/patch-mvninstall-hcatalog_server-extensions.txt
|
| mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/patch-mvninstall-itests_hive-unit.txt
|
| compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/patch-compile-hcatalog_server-extensions.txt
|
| compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/patch-compile-itests_hive-unit.txt
|
| javac | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/patch-compile-hcatalog_server-extensions.txt
|
| javac | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/patch-compile-itests_hive-unit.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/patch-findbugs-hcatalog_server-extensions.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus/patch-findbugs-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-common common standalone-metastore/metastore-server
hcatalog/server-extensions itests/hive-unit itests/util U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17246/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Remove periodical full refresh in HMS cache
> -------------------------------------------
>
>                 Key: HIVE-21697
>                 URL: https://issues.apache.org/jira/browse/HIVE-21697
>             Project: Hive
>          Issue Type: Improvement
>          Components: Standalone Metastore
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>            Priority: Major
>         Attachments: HIVE-21697.1.patch, HIVE-21697.2.patch, HIVE-21697.3.patch, HIVE-21697.4.patch,
HIVE-21697.5.patch
>
>
> In HIVE-18661, we added periodical notification based refresh in HMS cache. We shall
remove periodical full refresh to simplify the code as it will no longer be used. In the mean
time, we introduced mechanism to provide monotonic reads through the CachedStore.commitTransaction.
This will no longer be needed after HIVE-21637. So I will remove related code as well. This
will provide some performance benefits include:
> 1. We don't have to slow down write to catch up notification logs. Write can be done
immediately and tag the cache with writeids
> 2. We can read from cache even if updateUsingNotificationEvents is running. Read will
compare the writeids of the cache so monotonic reads will be guaranteed
> I'd like to put a patch separately with HIVE-21637 so it can be tested independently.
HMW will use periodical notification based refresh to update cache. And it will temporarily
lift the monotonic reads guarantee until HIVE-21637 checkin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message