spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng Lian (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-6456) Spark Sql throwing exception on large partitioned data
Date Mon, 23 Mar 2015 07:22:11 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375481#comment-14375481
] 

Cheng Lian edited comment on SPARK-6456 at 3/23/15 7:21 AM:
------------------------------------------------------------

How many partitions are there? Also, what's the version of the Hive metastore? For now, Spark
SQL only support Hive 0.12.0 and 0.13.1. Spark 1.1 and prior versions only support Hive 0.12.0.


was (Author: lian cheng):
How many partitions are there?

> Spark Sql throwing exception on large partitioned data
> ------------------------------------------------------
>
>                 Key: SPARK-6456
>                 URL: https://issues.apache.org/jira/browse/SPARK-6456
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: pankaj
>             Fix For: 1.2.1
>
>
> Spark connects with Hive Metastore. I am able to run simple queries like show table and
select. but throws below exception while running query on the hive Table having large number
of partitions.
> {noformat}
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
>         at`enter code here` org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
>         at org.apache.hadoop.hive.ql.metadata.Hive.getAllPartitionsOf(Hive.java:1785)
>         at org.apache.spark.sql.hive.HiveShim$.getAllPartitionsOf(Shim13.scala:316)
>         at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:86)
>         at org.apache.spark.sql.hive.HiveContext$$anon$1.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HiveContext.scala:253)
>         at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:137)
>         at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:137)
>         at scala.Option.getOrElse(Option.scala:120)
>         at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:137)
>         at org.apache.spark.sql.hive.HiveContext$$anon$1.lookupRelation(HiveContext.scala:253)
>         at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$5.applyOrElse(Analyzer.scala:143)
>         at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$5.applyOrElse(Analyzer.scala:138)
>         at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:144)
>         at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:162)
>         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>         at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
>         at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message