spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Maughan <davidamaug...@gmail.com>
Subject Re: spark 1.6.0 connect to hive metastore
Date Wed, 09 Mar 2016 17:00:05 GMT
Hi,

We're having a similar issue. We have a standalone cluster running 1.5.2
with Hive working fine having dropped hive-site.xml into the conf folder.
We've just updated to 1.6.0, using the same configuration. Now when
starting a spark-shell we get the following:

java.lang.RuntimeException: java.lang.RuntimeException: Unable to
instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreCli
ent
        at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at
org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)
        at
org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
        at
org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
        at
org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
        at
org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
        at
org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
        at
org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
        at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
        at
org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
        at
org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at
org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
        at $iwC$$iwC.<init>(<console>:15)
        at $iwC.<init>(<console>:24)
        at <init>(<console>:26)

On stepping though the code and enabling debug it shows that
hive.metastore.uris is not set:

DEBUG ClientWrapper: Hive Config: hive.metastore.uris=

..So it looks like it's not finding hive-site.xml? Weirdly, if I remove
hive-site.xml the exception does not occur which implies that it WAS on the
classpath...

Dave



On Tue, 9 Feb 2016 at 22:26 Koert Kuipers <koert@tresata.com> wrote:

> i do not have phoenix, but i wonder if its something related. will check
> my classpaths
>
> On Tue, Feb 9, 2016 at 5:00 PM, Benjamin Kim <bbuild11@gmail.com> wrote:
>
>> I got the same problem when I added the Phoenix plugin jar in the driver
>> and executor extra classpaths. Do you have those set too?
>>
>
>> On Feb 9, 2016, at 1:12 PM, Koert Kuipers <koert@tresata.com> wrote:
>>
>> yes its not using derby i think: i can see the tables in my actual hive
>> metastore.
>>
>> i was using a symlink to /etc/hive/conf/hive-site.xml for my
>> hive-site.xml which has a lot more stuff than just hive.metastore.uris
>>
>> let me try your approach
>>
>>
>>
>> On Tue, Feb 9, 2016 at 3:57 PM, Alexandr Dzhagriev <dzhagr@gmail.com>
>> wrote:
>>
>>> I'm using spark 1.6.0, hive 1.2.1 and there is just one property in the
>>> hive-site.xml hive.metastore.uris Works for me. Can you check in the
>>> logs, that when the HiveContext is created it connects to the correct uri
>>> and doesn't use derby.
>>>
>>> Cheers, Alex.
>>>
>>> On Tue, Feb 9, 2016 at 9:39 PM, Koert Kuipers <koert@tresata.com> wrote:
>>>
>>>> hey thanks. hive-site is on classpath in conf directory
>>>>
>>>> i currently got it to work by changing this hive setting in
>>>> hive-site.xml:
>>>> hive.metastore.schema.verification=true
>>>> to
>>>> hive.metastore.schema.verification=false
>>>>
>>>> this feels like a hack, because schema verification is a good thing i
>>>> would assume?
>>>>
>>>> On Tue, Feb 9, 2016 at 3:25 PM, Alexandr Dzhagriev <dzhagr@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Koert,
>>>>>
>>>>> As far as I can see you are using derby:
>>>>>
>>>>>  Using direct SQL, underlying DB is DERBY
>>>>>
>>>>> not mysql, which is used for the metastore. That means, spark couldn't
>>>>> find hive-site.xml on your classpath. Can you check that, please?
>>>>>
>>>>> Thanks, Alex.
>>>>>
>>>>> On Tue, Feb 9, 2016 at 8:58 PM, Koert Kuipers <koert@tresata.com>
>>>>> wrote:
>>>>>
>>>>>> has anyone successfully connected to hive metastore using spark
>>>>>> 1.6.0? i am having no luck. worked fine with spark 1.5.1 for me.
i am on
>>>>>> cdh 5.5 and launching spark with yarn.
>>>>>>
>>>>>> this is what i see in logs:
>>>>>> 16/02/09 14:49:12 INFO hive.metastore: Trying to connect to metastore
>>>>>> with URI thrift://metastore.mycompany.com:9083
>>>>>> 16/02/09 14:49:12 INFO hive.metastore: Connected to metastore.
>>>>>>
>>>>>> and then a little later:
>>>>>>
>>>>>> 16/02/09 14:49:34 INFO hive.HiveContext: Initializing execution hive,
>>>>>> version 1.2.1
>>>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Inspected Hadoop
>>>>>> version: 2.6.0-cdh5.4.4
>>>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Loaded
>>>>>> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.4.4
>>>>>> 16/02/09 14:49:34 WARN conf.HiveConf: HiveConf of name
>>>>>> hive.server2.enable.impersonation does not exist
>>>>>> 16/02/09 14:49:35 INFO metastore.HiveMetaStore: 0: Opening raw store
>>>>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>>>>>> 16/02/09 14:49:35 INFO metastore.ObjectStore: ObjectStore, initialize
>>>>>> called
>>>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property
>>>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored
>>>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property
>>>>>> datanucleus.cache.level2 unknown - will be ignored
>>>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified but
>>>>>> not present in CLASSPATH (or one of dependencies)
>>>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified but
>>>>>> not present in CLASSPATH (or one of dependencies)
>>>>>> 16/02/09 14:49:37 WARN conf.HiveConf: HiveConf of name
>>>>>> hive.server2.enable.impersonation does not exist
>>>>>> 16/02/09 14:49:37 INFO metastore.ObjectStore: Setting MetaStore
>>>>>> object pin classes with
>>>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
>>>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class
>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class
>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class
>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class
>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>>>>>> "embedded-only" so does not have its own datastore table.
>>>>>> 16/02/09 14:49:40 INFO metastore.MetaStoreDirectSql: Using direct
>>>>>> SQL, underlying DB is DERBY
>>>>>> 16/02/09 14:49:40 INFO metastore.ObjectStore: Initialized ObjectStore
>>>>>> java.lang.RuntimeException: java.lang.RuntimeException: Unable to
>>>>>> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>>>>>>   at
>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
>>>>>>   at
>>>>>> org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)
>>>>>>   at
>>>>>> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
>>>>>>   at
>>>>>> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
>>>>>>   at
>>>>>> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
>>>>>>   at
>>>>>> org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:440)
>>>>>>   at
>>>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272)
>>>>>>   at
>>>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:271)
>>>>>>   at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>>>>>>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>>>>>>   at
>>>>>> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>>>>>>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>>>>>>   at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:271)
>>>>>>   at
>>>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:97)
>>>>>>   at
>>>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
>>>>>>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>> Method)
>>>>>>   at
>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>>   at
>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>>   at org.apache.spark.repl.Main$.createSQLContext(Main.scala:89)
>>>>>>   ... 47 elided
>>>>>> Caused by: java.lang.RuntimeException: Unable to instantiate
>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
>>>>>>   at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
>>>>>>   ... 66 more
>>>>>> Caused by: java.lang.reflect.InvocationTargetException:
>>>>>> org.apache.hadoop.hive.metastore.api.MetaException: Version information
not
>>>>>> found in metastore.
>>>>>>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>> Method)
>>>>>>   at
>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>>   at
>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
>>>>>>   ... 72 more
>>>>>> Caused by: org.apache.hadoop.hive.metastore.api.MetaException:
>>>>>> Version information not found in metastore.
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6664)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)
>>>>>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>   at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>   at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>   at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
>>>>>>   at com.sun.proxy.$Proxy26.verifySchema(Unknown Source)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
>>>>>>   at
>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
>>>>>>   ... 77 more
>>>>>>
>>>>>> i seem to get this exact same error no matter what i set
>>>>>> spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars
to. i
>>>>>> tried building different spark 1.6.0 jars (with hadoop provided,
with
>>>>>> hadoop included), but no effect.
>>>>>>
>>>>>> any ideas?
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>

Mime
View raw message