hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vihang Karajgaonkar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
Date Thu, 16 Aug 2018 14:58:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582641#comment-16582641
] 

Vihang Karajgaonkar commented on HIVE-20401:
--------------------------------------------

Looks like the second thread acquires the compile lock and is waiting from HDFS side to find
out if that path is encrypted location or not. Do you see any errors in the namenode logs?

> HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
> --------------------------------------------------------------------------------------------
>
>                 Key: HIVE-20401
>                 URL: https://issues.apache.org/jira/browse/HIVE-20401
>             Project: Hive
>          Issue Type: Bug
>          Components: Beeline, HiveServer2
>    Affects Versions: 1.2.2
>            Reporter: zhangbutao
>            Priority: Major
>         Attachments: HiveServer2_jstack.txt
>
>
> HiveServer2  process often gets stuck , and jstack shows about one hundred thread is
blocked at the following code,waiting the resource *0x00000003c0acca40*
> {code:java}
> "HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0 tid=0x00007fd71470c000
nid=0x3e1f waiting for monitor entry [0x00007fd6d6eee000]
> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
> 	- waiting to lock <0x00000003c0acca40> (a java.lang.Object)
> 	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
> 	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> 	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
> 	at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> 	at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
> 	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
> 	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> {code}
>  We find one thread locked the resource *0x00000003c0acca40*: ,and the jstack is as follow:
> {code:java}
> "HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0 tid=0x00007fd71433e000
nid=0x37c0 in Object.wait() [0x00007fd6d04b8000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
> 	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> 	- locked <0x00000003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1351)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> 	at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
> 	at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> 	- locked <0x00000003d9ab5dc8> (a org.apache.hadoop.io.retry.RetryInvocationHandler$Call)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
> 	at com.sun.proxy.$Proxy19.getEZForPath(Unknown Source)
> 	at org.apache.hadoop.hdfs.DFSClient.getEZForPath(DFSClient.java:2810)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$51.doCall(DistributedFileSystem.java:2270)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$51.doCall(DistributedFileSystem.java:2267)
> 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getEZForPath(DistributedFileSystem.java:2286)
> 	at org.apache.hadoop.hdfs.client.HdfsAdmin.getEncryptionZoneForPath(HdfsAdmin.java:350)
> 	at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.isPathEncrypted(Hadoop23Shims.java:1221)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:1883)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:1964)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStagingDirectoryPathname(SemanticAnalyzer.java:1996)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1809)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1544)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10076)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10127)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:210)
> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:425)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:309)
> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1145)
> 	- locked <0x00000003c0acca40> (a java.lang.Object)
> 	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
> 	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> 	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
> 	at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> 	at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
> 	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
> 	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}
> It seems that Thread-809399 which locked the resource *0x00000003c0acca40* is waiting
for rpc responseļ¼Œ leading to other threads are blocked. Does anyone know what caused this?
  Hadoop version is 2.8.2.  Attachment is full jstack.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message