hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20442) Hive stale lock when the hiveserver2 background thread died with NPE
Date Fri, 26 Jul 2019 19:41:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894077#comment-16894077
] 

Hive QA commented on HIVE-20442:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12975940/HIVE-20442.01.branch-2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18175/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18175/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18175/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ date '+%Y-%m-%d %T.%3N'
2019-07-26 19:38:49.300
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18175/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-2 ]]
+ [[ -d apache-github-branch-2-source ]]
+ [[ ! -d apache-github-branch-2-source/.git ]]
+ [[ ! -d apache-github-branch-2-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-07-26 19:38:49.383
+ cd apache-github-branch-2-source
+ git fetch origin
>From https://github.com/apache/hive
   fd2f7c8..2039350  branch-2   -> origin/branch-2
   7fecb6f..7534f82  branch-1   -> origin/branch-1
   93163cb..9fb2238  branch-2.3 -> origin/branch-2.3
   31a417e..91c243c  branch-3   -> origin/branch-3
   3e16420..378083e  branch-3.1 -> origin/branch-3.1
   5116db0..4f7feaa  master     -> origin/master
 * [new branch]      revert-648-hive21783 -> origin/revert-648-hive21783
+ git reset --hard HEAD
HEAD is now at fd2f7c8 HIVE-21484 : Metastore API getVersion() should return real version
(Vihang Karajgaonkar, reviewed by Naveen Gangam and Peter Vary)
+ git clean -f -d
+ git checkout branch-2
Already on 'branch-2'
Your branch is behind 'origin/branch-2' by 9 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/branch-2
HEAD is now at 2039350 HIVE-21786 Update repo URLs in poms branch 2.3 version
+ git merge --ff-only origin/branch-2
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-07-26 19:39:12.330
+ rm -rf ../yetus_PreCommit-HIVE-Build-18175
+ mkdir ../yetus_PreCommit-HIVE-Build-18175
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18175
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18175/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java: does not exist in index
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:1482
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java' with conflicts.
Going to apply patch with: git apply -p1
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:1482
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java' with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/Driver.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-18175
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12975940 - PreCommit-HIVE-Build

> Hive stale lock when the hiveserver2 background thread died with NPE
> --------------------------------------------------------------------
>
>                 Key: HIVE-20442
>                 URL: https://issues.apache.org/jira/browse/HIVE-20442
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive, Transactions
>    Affects Versions: 2.1.1
>         Environment: Hive-2.1
>            Reporter: Rajkumar Singh
>            Assignee: Rajkumar Singh
>            Priority: Major
>         Attachments: HIVE-20442.01.branch-2.patch
>
>
> this look like a race condition where background thread is not able to release the lock
it aquired.
> 1. hiveserver2 background thread request for lock
> {code}
> 2018-08-20T14:13:38,813 INFO  [HiveServer2-Background-Pool: Thread-XXXXX]: lockmgr.DbLockManager
(DbLockManager.java:lock(100)) - Requesting: queryId=hive_xxxxxxx LockRequest(component:[LockComponent(type:SHARED_READ,
level:TABLE, dbname:testdb, tablename:test_table, operationType:SELECT)], txnid:0, user:hive,
hostname:HOSTNAME, agentInfo:hive_xxxxxxx)
> {code}
> 2. acquired the lock and start heartbeating
> {code}
> 2018-08-20T14:36:30,233 INFO  [HiveServer2-Background-Pool: Thread-XXXXX]: lockmgr.DbTxnManager
(DbTxnManager.java:startHeartbeat(517)) - Started heartbeat with delay/interval = 150000/150000
            MILLISECONDS for query: agentInfo:hive_xxxxxxx
> {code}
> 3. during time between event #1 and #2, client disconnected and deleteContext cleanup
the session dir
> {code}
> 2018-08-21T15:39:57,820 INFO  [HiveServer2-Handler-Pool: Thread-XXX]: thrift.ThriftCLIService
(ThriftBinaryCLIService.java:deleteContext(136)) - Session disconnected without closing properly.
> 2018-08-21T15:39:57,820 INFO  [HiveServer2-Handler-Pool: Thread-XXXX]: thrift.ThriftCLIService
(ThriftBinaryCLIService.java:deleteContext(140)) - Closing the session: SessionHandle [3be07faf-5544-4178-8b50-8173002b171a]
> 2018-08-21T15:39:57,820 INFO  [HiveServer2-Handler-Pool: Thread-XXXX]: service.CompositeService
(SessionManager.java:closeSession(363)) - Session closed, SessionHandle [xxxxxxxxxxxxxxxxxxxxxxx],
current sessions:2
> {code}
> 4. background thread died with NPE while trying to get the queryid 
> {code}
> java.lang.NullPointerException: null
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1568) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204) ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
>         at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
[hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
>         at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
[hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
>         at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
[hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
>         at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_77]
>         at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_77]
> {code}
> did not get a chance to release the lock and heartbeater thread continue heartbeat indefinately.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message