spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 诺铁 <noty...@gmail.com>
Subject query over hive context hangs, please help
Date Wed, 22 Jul 2015 03:08:57 GMT
The thread dump is here, seems hang on accessing mysql meta store.
I googled and find a bug related to com.mysql.jdbc.util.ReadAheadInputStream,
but don't have a workaround.
And I am not sure about that.   please help me. thanks.

thread dump------->
"MyAppDefaultScheduler_Worker-2" prio=10 tid=0x00007f5e50463000 nid=0xd17f
runnable [0x00007f5e06ff9000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at
com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:156)

        at
com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)

        - locked <0x000000060b6aaba0> (a
com.mysql.jdbc.util.ReadAheadInputStream)
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3671)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4149)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2834)
        - locked <0x000000060b6493a8> (a com.mysql.jdbc.JDBC4Connection)
        at
com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:5200)
        at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5083)
        - locked <0x000000060b6493a8> (a com.mysql.jdbc.JDBC4Connection)
        at
com.jolbox.bonecp.ConnectionHandle.rollback(ConnectionHandle.java:1272)
        at
org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource.rollback(ConnectionFactoryImpl.java:705)

        at
org.datanucleus.transaction.Transaction.rollback(Transaction.java:548)
        at
org.datanucleus.transaction.TransactionManager.rollback(TransactionManager.java:85)

        at
org.datanucleus.TransactionImpl.internalRollback(TransactionImpl.java:517)
        - locked <0x0000000614b7e3c0> (a org.datanucleus.TransactionImpl)
        at
org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:445)
        at
org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:182)
        at
org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:438)

        at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.close(ObjectStore.java:2278)

        at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2225)

        at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1559)

        at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1553)

        at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)

        at com.sun.proxy.$Proxy20.getPartitions(Unknown Source)
        at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:2516)

        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)

        at com.sun.proxy.$Proxy21.get_partitions(Unknown Source)
        at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:856)

        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)

        at com.sun.proxy.$Proxy22.listPartitions(Unknown Source)
        at
org.apache.hadoop.hive.ql.metadata.Hive.getAllPartitionsOf(Hive.java:1782)
        at
org.apache.spark.sql.hive.HiveShim$.getAllPartitionsOf(Shim13.scala:354)
        at
org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:226)

        - locked <0x0000000619304820> (a
org.apache.spark.sql.hive.HiveContext$$anon$3)
        at org.apache.spark.sql.hive.HiveContext$$anon$3.org
$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HiveContext.scala:262)

        at
org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:161)

        at
org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:161)

        at scala.Option.getOrElse(Option.scala:120)
        at
org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:161)

        at
org.apache.spark.sql.hive.HiveContext$$anon$3.lookupRelation(HiveContext.scala:262)

        at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:174)

        at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$6.applyOrElse(Analyzer.scala:186)

        at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$6.applyOrElse(Analyzer.scala:181)

        at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:188)

        at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:188)

        at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:51)

        at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:187)

        at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:208)

        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
        at scala.collection.TraversableOnce$class.to
(TraversableOnce.scala:273)
        at scala.collection.AbstractIterator.to(Iterator.scala:1157)
        at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
        at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
        at
org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:238)

        at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:193)

        at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:208)

        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
        at scala.collection.TraversableOnce$class.to
(TraversableOnce.scala:273)
        at scala.collection.AbstractIterator.to(Iterator.scala:1157)
        at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
        at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
        at
org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:238)

        at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:193)

        at
org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:178)
        at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:181)

        at
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:171)

        at
org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:61)

        at
org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:59)

        at
scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)

        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at
org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:59)

        at
org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:51)

        at scala.collection.immutable.List.foreach(List.scala:318)
        at
org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:51)

        at
org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:1082)

        - locked <0x000000061ec26458> (a
org.apache.spark.sql.hive.HiveContext$QueryExecution)
        at
org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:1082)

        at
org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:1080)

        at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
        at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
        at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:101)

Mime
View raw message