hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-23439) Hive sessions over 24 hours encounter Kerberos-related StatsTask errors
Date Tue, 02 Jun 2020 04:20:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-23439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123336#comment-17123336
] 

Hive QA commented on HIVE-23439:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m  9s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  3s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 53s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 48s{color} | {color:blue}
ql in master has 1524 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 57s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 25s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  1s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  1s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 50s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 55s{color} | {color:red}
ql generated 1 new + 1524 unchanged - 0 fixed = 1525 total (was 1524) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 56s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 13s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 43s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Class org.apache.hadoop.hive.ql.exec.StatsTask defines non-transient non-serializable
instance field dbb  In StatsTask.java:instance field dbb  In StatsTask.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-22706/dev-support/hive-personality.sh
|
| git revision | master / b45e152 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-22706/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-22706/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Hive sessions over 24 hours encounter Kerberos-related StatsTask errors
> -----------------------------------------------------------------------
>
>                 Key: HIVE-23439
>                 URL: https://issues.apache.org/jira/browse/HIVE-23439
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2, Standalone Metastore
>    Affects Versions: 3.1.0
>            Reporter: Chiran Ravani
>            Assignee: Rajkumar Singh
>            Priority: Critical
>         Attachments: HIVE-23439.patch
>
>
> We have an application that uses Hive via JDBC. The interesting thing about them is that
they have sessions that are established with HiveServer2 for multiple days. After 24 hours,
their queries are failing with StatsTask-related errors. From looking in the logs, it looks
like the communication breaks down between HiveServer2 and the MetaStore.
> Below is error seen:
> {code}
> 2020-04-22T21:25:53,248 ERROR [Thread-1202599]: exec.StatsTask (:()) - Failed to run
stats task
> org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException:
Unable to fetch table tennis. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>  at org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4927)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:189)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:86)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:108) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:82) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table tennis.
Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1387) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  ... 6 more
> Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>  at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4790) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4858) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4838) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1378) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1336) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1316) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1298) ~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  at org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4918)
~[hive-exec-3.1.0.3.1.4.39-6.jar:3.1.0.3.1.4.39-6]
>  ... 6 more
> {code}
> The problem appears to be because of delegation token issued by Hive Metastore could
not be renewed by HiveServer2 within 24 hours period of time.
> There is similar issue reported in upstream HIVE-22033 which adderesses similar kind
of issue, I backported that fix on my local cluster and deployed the same, but that does not
seems to adderess the issue. Problem seems to be GC interval which removed th DT from HMS
after expiry which by default is set to 1 hour.
> "hive.cluster.delegation.token.gc-interval", 1, TimeUnit.HOURS, "")



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message