hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sungpeo Kook (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8519) Hive metastore lock wait timeout
Date Thu, 02 Jan 2020 10:06:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006712#comment-17006712
] 

Sungpeo Kook commented on HIVE-8519:
------------------------------------

This error is caused by the difference in values ​​for *hive.metastore.client.socket.timeout*
which is set to hive-site.xml.

The value for the property is used by hiveserver2 and hivemetastore.

But as you know, you can set properties differently for each service. In this case, hiveserver2
and hivemetastore can have different value for the same proeprty.

 

Above situation occured when hiveserver2's timeout < hivemetastore's timeout.

 

Supposed, you are trying to drop table which includes many partitions. It might be taking
long time.
 # hiveserver2 request to drop table
 # hivemetastore start to drop table including partitions in a transaction.
 # After hive.metastore.client.socket.timeout, hiveserver2 got timeout exception and retry
to drop table again.
 # hivemetastore is still doing drop table which is requested before, and start to drop table
again.
 # New thread for dropping table get lock wait timeout exception.

 

+If you set hive.metastore.client.socket.timeout to same value for hiveserver2 and hivemetastore,
you wouldn't face this issue.+

> Hive metastore lock wait timeout
> --------------------------------
>
>                 Key: HIVE-8519
>                 URL: https://issues.apache.org/jira/browse/HIVE-8519
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 0.10.0
>            Reporter: Jason Liao
>            Assignee: Sungpeo Kook
>            Priority: Major
>
> We got a lot of exception as below when doing a drop table partition, which made hive
query every every slow. For example, it will cost 250s while executing use db_test;
> Log:
> 2014-10-17 04:04:46,873 ERROR Datastore.Persist (Log4JLogger.java:error(115)) - Update
of object "org.apache.hadoop.hive.metastore.model.MStorageDescriptor@13c9c4b3" using statement
"UPDATE `SDS` SET `CD_ID`=? WHERE `SD_ID`=?" failed : java.sql.SQLException: Lock wait timeout
exceeded; try restarting transaction
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4096)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4028)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
>         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2734)
>         at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2458)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2375)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2359)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>         at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeUpdate(ParamLoggingPreparedStatement.java:399)
>         at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:439)
>         at org.datanucleus.store.rdbms.request.UpdateRequest.execute(UpdateRequest.java:374)
>         at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateTable(RDBMSPersistenceHandler.java:417)
>         at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateObject(RDBMSPersistenceHandler.java:390)
>         at org.datanucleus.state.JDOStateManager.flush(JDOStateManager.java:5012)
>         at org.datanucleus.FlushOrdered.execute(FlushOrdered.java:106)
>         at org.datanucleus.ExecutionContextImpl.flushInternal(ExecutionContextImpl.java:4019)
>         at org.datanucleus.ExecutionContextThreadedImpl.flushInternal(ExecutionContextThreadedImpl.java:450)
>         at org.datanucleus.store.query.Query.prepareDatastore(Query.java:1575)
>         at org.datanucleus.store.query.Query.executeQuery(Query.java:1760)
>         at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
>         at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:243)
>         at org.apache.hadoop.hive.metastore.ObjectStore.listStorageDescriptorsWithCD(ObjectStore.java:2185)
>         at org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:2131)
>         at org.apache.hadoop.hive.metastore.ObjectStore.preDropStorageDescriptor(ObjectStore.java:2162)
>         at org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionCommon(ObjectStore.java:1361)
>         at org.apache.hadoop.hive.metastore.ObjectStore.dropPartition(ObjectStore.java:1301)
>         at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
>         at $Proxy4.dropPartition(Unknown Source)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partition_common(HiveMetaStore.java:1865)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partition(HiveMetaStore.java:1911)
>         at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>         at $Proxy5.drop_partition(Unknown Source)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message