spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Reina <alex.k...@gmail.com>
Subject Re: Save hive table from spark in hive 2.1.0
Date Sun, 10 Dec 2017 11:17:44 GMT
I have tried what you propose, added the property to hive-site.xml, and
although with this option I can run hive, this does not solve my problem.
I'm sorry if maybe you explain me wrongly.

I need to save a dataframe transformed into spark in hive, with the version
of scheme 2.1.1 of hive (last stable version of hive until 2 months ago), by
default spark internally uses version 1.2.1 and this generates conflict with
my other applications the ecosystem.

This hiveTable created from spark, is not visible from any other application
due to its version of the metadata, so I need to create from spark a hive
table in its corresponding version.

Hive schematool
root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -info --verbose
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://localhost/metastore?useSSL=false
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 hive
Hive distribution version:	 2.1.0
Metastore schema version:	 1.2.0
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema version
is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema version
is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0
	at
org.apache.hive.beeline.HiveSchemaTool.assertCompatibleVersion(HiveSchemaTool.java:202)
	at org.apache.hive.beeline.HiveSchemaTool.showInfo(HiveSchemaTool.java:139)
	at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:498)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
*** schemaTool failed ***

If i try upgrade schema too have error.
root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -upgradeSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://localhost/metastore?useSSL=false
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 hive
Starting upgrade metastore schema from version 1.2.0 to 2.1.0
Upgrade script upgrade-1.2.0-to-2.0.0.mysql.sql
Error: Duplicate column name 'CQ_HIGHEST_TXN_ID' (state=42S21,code=1060)
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED!
Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
Use --verbose for detailed stacktrace.
*** schemaTool failed ***

thank you very much for your time dedicated to helping me




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message