spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Campagnola, Francesco" <Francesco.Campagn...@anritsu.com>
Subject Spark 2.0.0 Thrift Server problem with Hive metastore
Date Mon, 05 Sep 2016 15:25:31 GMT
Hi,

in an already working Spark - Hive environment with Spark 1.6 and Hive 1.2.1, with Hive metastore
configured on Postgres DB, I have upgraded Spark to the 2.0.0.

I have started the thrift server on YARN, then tried to execute from the beeline cli or a
jdbc client the following command:
SHOW DATABASES;
It always gives this error on Spark server side:

spark@spark-test[spark] /home/spark> beeline -u jdbc:hive2://$(hostname):10000 -n spark

Connecting to jdbc:hive2://spark-test:10000
16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:10000
16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:10000
16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri:
jdbc:hive2:// spark-test:10000
Connected to: Spark SQL (version 2.0.0)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive

0: jdbc:hive2:// spark-test:10000> show databases;
java.lang.IllegalStateException: Can't overwrite cause with java.lang.ClassCastException:
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
        at java.lang.Throwable.initCause(Throwable.java:457)
        at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
        at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
        at org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:197)
        at org.apache.hive.service.cli.HiveSQLException.<init>(HiveSQLException.java:108)
        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
        at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365)
        at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:42)
        at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
        at org.apache.hive.beeline.Commands.execute(Commands.java:860)
        at org.apache.hive.beeline.Commands.sql(Commands.java:713)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage
3.0 failed 10 times, most recent failure: Lost task 0.9 in stage 3.0 (TID 12, vertica204):
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow
cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:247)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
        at org.apache.spark.scheduler.Task.run(Task.scala:85)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:244)
        at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:210)
        ... 15 more
Error: Error retrieving next row (state=,code=0)

The same command works when using Spark 1.6, is it a possible issue?

Thanks!

Mime
View raw message