spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ferran Galí <ferrang...@trovit.com>
Subject Troubles interacting with different version of Hive metastore
Date Fri, 25 Sep 2015 11:02:08 GMT
Hello,

I'm trying to start the SparkSQL thriftserver over YARN, connecting it to
the 1.1.0-cdh5.4.3 hive metastore that we already have in production.

I downloaded the latest version of Spark (1.5.0), I just followed the
instructions from the documentation
<http://spark.apache.org/docs/latest/sql-programming-guide.html#interacting-with-different-versions-of-hive-metastore>,
and I configured the spark-default.conf file with the following parameters:

*spark.sql.hive.metastore.version* 1.1.0
*spark.sql.hive.metastore.jars*
 {path-to-hive-install}/lib/*:{path-to-hadoop-install}/common/*:{path-to-hadoop-install}{path-to-hadoop-install}/hdfs/*:{path-to-hadoop-install}/hdfs/lib/*:{path-to-hadoop-install}/yarn/*:{path-to-hadoop-install}/yarn/lib/*:{path-to-hadoop-install}/mapreduce/*:{path-to-hadoop-install}/mapreduce/lib/*


Moreover, I've also placed the *hive-site.xml* into the *conf* folder.
When I startup the thriftserver, it eventually crashes with the following
message:

15/09/25 12:50:57 INFO hive.HiveContext: default warehouse location is
/user/hive/warehouse
15/09/25 12:50:57 INFO hive.HiveContext: Initializing
HiveMetastoreConnection version 1.1.0 using [Ljava.net.URL;@51fd1e7f
15/09/25 12:50:58 INFO client.ClientWrapper: Inspected Hadoop version:
2.6.0-cdh5.4.3
Exception in thread "main" java.lang.ClassNotFoundException:
java.lang.NoClassDefFoundError: com/google/common/base/Predicate when
creating Hive client using classpath: *HERE ALL THE JARS INSIDE THE
**spark.sql.hive.metastore.jars
(including guava-14.0.1.jar)*


If I put "maven" inside the *spark.sql.hive.metastore.jars* parameter I get
the same error.

Do you know what could I be doing wrong? I don't understand, because the
class that he's complaining (com.google.common.base.Predicate) it's already
in the classpath.

Yours,
Ferran Galí i Reniu
-- 
[image: Trovit] <http://www.trovit.com/>
[image: Twitter] <http://twitter.com/trovit>[image: Facebook]
<http://www.facebook.com/trovit.search>[image: Linkedin]
<http://www.linkedin.com/company/trovit>[image: Google +]
<http://plus.google.com/+trovit/>[image: Blog]
<http://about.trovit.com/blog/>
*Ferran Galí i Reniu*
Pipeline
+34 93 209 2556
Avda. Diagonal 601, 9ª
08028, Barcelona

Mime
View raw message