spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suniti Singh <suniti.si...@gmail.com>
Subject Re: spark 1.6.0 connect to hive metastore
Date Wed, 09 Mar 2016 18:48:02 GMT
Hi,

I am able to reproduce this error only when using spark 1.6.0 and hive
1.6.0. The hive-site.xml is in the classpath but somehow spark rejects the
classpath search for hive-site.xml and start using the default metastore
Derby.

16/03/09 10:37:52 INFO MetaStoreDirectSql: Using direct SQL, underlying DB
is DERBY

16/03/09 10:37:52 INFO ObjectStore: Initialized ObjectStore

16/03/09 10:37:52 WARN Hive: Failed to access metastore. This class should
not accessed in runtime.

org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)

at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)

at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166)

at org.apache.hadoop.hive.ql.session.SessionState.start(
SessionState.java:503)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(
ClientWrapper.scala:194)

On Wed, Mar 9, 2016 at 9:00 AM, Dave Maughan <davidamaughan@gmail.com>
wrote:

> Hi,
>
> We're having a similar issue. We have a standalone cluster running 1.5.2
> with Hive working fine having dropped hive-site.xml into the conf folder.
> We've just updated to 1.6.0, using the same configuration. Now when
> starting a spark-shell we get the following:
>
> java.lang.RuntimeException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreCli
> ent
>         at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
>         at
> org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)
>         at
> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
>         at
> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
>         at
> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
>         at
> org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
>         at
> org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
>         at
> org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
>         at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
>         at
> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
>         at
> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>         at
> org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
>         at $iwC$$iwC.<init>(<console>:15)
>         at $iwC.<init>(<console>:24)
>         at <init>(<console>:26)
>
> On stepping though the code and enabling debug it shows that
> hive.metastore.uris is not set:
>
> DEBUG ClientWrapper: Hive Config: hive.metastore.uris=
>
> ..So it looks like it's not finding hive-site.xml? Weirdly, if I remove
> hive-site.xml the exception does not occur which implies that it WAS on the
> classpath...
>
> Dave
>
>
>
> On Tue, 9 Feb 2016 at 22:26 Koert Kuipers <koert@tresata.com> wrote:
>
>> i do not have phoenix, but i wonder if its something related. will check
>> my classpaths
>>
>> On Tue, Feb 9, 2016 at 5:00 PM, Benjamin Kim <bbuild11@gmail.com> wrote:
>>
>>> I got the same problem when I added the Phoenix plugin jar in the driver
>>> and executor extra classpaths. Do you have those set too?
>>>
>>
>>> On Feb 9, 2016, at 1:12 PM, Koert Kuipers <koert@tresata.com> wrote:
>>>
>>> yes its not using derby i think: i can see the tables in my actual hive
>>> metastore.
>>>
>>> i was using a symlink to /etc/hive/conf/hive-site.xml for my
>>> hive-site.xml which has a lot more stuff than just hive.metastore.uris
>>>
>>> let me try your approach
>>>
>>>
>>>
>>> On Tue, Feb 9, 2016 at 3:57 PM, Alexandr Dzhagriev <dzhagr@gmail.com>
>>> wrote:
>>>
>>>> I'm using spark 1.6.0, hive 1.2.1 and there is just one property in the
>>>> hive-site.xml hive.metastore.uris Works for me. Can you check in the
>>>> logs, that when the HiveContext is created it connects to the correct uri
>>>> and doesn't use derby.
>>>>
>>>> Cheers, Alex.
>>>>
>>>> On Tue, Feb 9, 2016 at 9:39 PM, Koert Kuipers <koert@tresata.com>
>>>> wrote:
>>>>
>>>>> hey thanks. hive-site is on classpath in conf directory
>>>>>
>>>>> i currently got it to work by changing this hive setting in
>>>>> hive-site.xml:
>>>>> hive.metastore.schema.verification=true
>>>>> to
>>>>> hive.metastore.schema.verification=false
>>>>>
>>>>> this feels like a hack, because schema verification is a good thing i
>>>>> would assume?
>>>>>
>>>>> On Tue, Feb 9, 2016 at 3:25 PM, Alexandr Dzhagriev <dzhagr@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Koert,
>>>>>>
>>>>>> As far as I can see you are using derby:
>>>>>>
>>>>>>  Using direct SQL, underlying DB is DERBY
>>>>>>
>>>>>> not mysql, which is used for the metastore. That means, spark
>>>>>> couldn't find hive-site.xml on your classpath. Can you check that,
please?
>>>>>>
>>>>>> Thanks, Alex.
>>>>>>
>>>>>> On Tue, Feb 9, 2016 at 8:58 PM, Koert Kuipers <koert@tresata.com>
>>>>>> wrote:
>>>>>>
>>>>>>> has anyone successfully connected to hive metastore using spark
>>>>>>> 1.6.0? i am having no luck. worked fine with spark 1.5.1 for
me. i am on
>>>>>>> cdh 5.5 and launching spark with yarn.
>>>>>>>
>>>>>>> this is what i see in logs:
>>>>>>> 16/02/09 14:49:12 INFO hive.metastore: Trying to connect to
>>>>>>> metastore with URI thrift://metastore.mycompany.com:9083
>>>>>>> 16/02/09 14:49:12 INFO hive.metastore: Connected to metastore.
>>>>>>>
>>>>>>> and then a little later:
>>>>>>>
>>>>>>> 16/02/09 14:49:34 INFO hive.HiveContext: Initializing execution
>>>>>>> hive, version 1.2.1
>>>>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Inspected Hadoop
>>>>>>> version: 2.6.0-cdh5.4.4
>>>>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Loaded
>>>>>>> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version
2.6.0-cdh5.4.4
>>>>>>> 16/02/09 14:49:34 WARN conf.HiveConf: HiveConf of name
>>>>>>> hive.server2.enable.impersonation does not exist
>>>>>>> 16/02/09 14:49:35 INFO metastore.HiveMetaStore: 0: Opening raw
store
>>>>>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>>>>>>> 16/02/09 14:49:35 INFO metastore.ObjectStore: ObjectStore,
>>>>>>> initialize called
>>>>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property
>>>>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored
>>>>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property
>>>>>>> datanucleus.cache.level2 unknown - will be ignored
>>>>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified
but
>>>>>>> not present in CLASSPATH (or one of dependencies)
>>>>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified
but
>>>>>>> not present in CLASSPATH (or one of dependencies)
>>>>>>> 16/02/09 14:49:37 WARN conf.HiveConf: HiveConf of name
>>>>>>> hive.server2.enable.impersonation does not exist
>>>>>>> 16/02/09 14:49:37 INFO metastore.ObjectStore: Setting MetaStore
>>>>>>> object pin classes with
>>>>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
>>>>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class
>>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged
as
>>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class
>>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class
>>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged
as
>>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class
>>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>>> 16/02/09 14:49:40 INFO metastore.MetaStoreDirectSql: Using direct
>>>>>>> SQL, underlying DB is DERBY
>>>>>>> 16/02/09 14:49:40 INFO metastore.ObjectStore: Initialized ObjectStore
>>>>>>> java.lang.RuntimeException: java.lang.RuntimeException: Unable
to
>>>>>>> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:440)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:271)
>>>>>>>   at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>>>>>>>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>>>>>>>   at
>>>>>>> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>>>>>>>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>>>>>>>   at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:271)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:97)
>>>>>>>   at
>>>>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
>>>>>>>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>> Method)
>>>>>>>   at
>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>>>   at
>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>>>   at org.apache.spark.repl.Main$.createSQLContext(Main.scala:89)
>>>>>>>   ... 47 elided
>>>>>>> Caused by: java.lang.RuntimeException: Unable to instantiate
>>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
>>>>>>>   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
>>>>>>>   ... 66 more
>>>>>>> Caused by: java.lang.reflect.InvocationTargetException:
>>>>>>> org.apache.hadoop.hive.metastore.api.MetaException: Version information
not
>>>>>>> found in metastore.
>>>>>>>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>> Method)
>>>>>>>   at
>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>>>   at
>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
>>>>>>>   ... 72 more
>>>>>>> Caused by: org.apache.hadoop.hive.metastore.api.MetaException:
>>>>>>> Version information not found in metastore.
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6664)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)
>>>>>>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>   at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>   at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>   at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
>>>>>>>   at com.sun.proxy.$Proxy26.verifySchema(Unknown Source)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
>>>>>>>   at
>>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
>>>>>>>   ... 77 more
>>>>>>>
>>>>>>> i seem to get this exact same error no matter what i set
>>>>>>> spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars
to. i
>>>>>>> tried building different spark 1.6.0 jars (with hadoop provided,
with
>>>>>>> hadoop included), but no effect.
>>>>>>>
>>>>>>> any ideas?
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>

Mime
View raw message