spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Girardot <o.girar...@lateral-thoughts.com>
Subject Re: Spark Shell Hive Context and Kerberos ticket
Date Fri, 26 Jun 2015 06:03:45 GMT
Nop I have not but I'm glad I'm not the only one :p

Le ven. 26 juin 2015 07:54, Tao Li <litao.buptsse@gmail.com> a écrit :

> Hi Olivier, have you fix this problem now? I still have this fasterxml
> NoSuchMethodError.
>
> 2015-06-18 3:08 GMT+08:00 Olivier Girardot <
> o.girardot@lateral-thoughts.com>:
>
>> Ok what was wrong was that the spark-env did not contain the
>> HADOOP_CONF_DIR properly set to /etc/hadoop/conf/
>> With that fixed, this issue is gone, but I can't seem to get Spark SQL
>> 1.4.0 with Hive working on CDH 5.3 or 5.4 :
>> Using this command line :
>> IPYTHON=1 /.../spark-1.4.0-bin-hadoop2.4/bin/pyspark  --master
>> yarn-client --driver-class-path `hadoop classpath`
>>
>> I end up with this issue :
>> : java.lang.NoSuchMethodError:
>> com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;
>> at
>> com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<init>(ScalaNumberDeserializersModule.scala:49)
>> at
>> com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<clinit>(ScalaNumberDeserializersModule.scala)
>> at
>> com.fasterxml.jackson.module.scala.deser.ScalaNumberDeserializersModule$class.$init$(ScalaNumberDeserializersModule.scala:61)
>> at
>> com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:19)
>> at
>> com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:35)
>> at
>> com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
>> at
>> org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:78)
>> at
>> org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87)
>> at
>> org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:118)
>> at
>> org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:125)
>> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1255)
>> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:1189)
>> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:1248)
>> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:176)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
>> at py4j.Gateway.invoke(Gateway.java:259)
>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>> at py4j.GatewayConnection.run(GatewayConnection.java:207)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> This seems to be related to this Jira Issue :
>> https://issues.apache.org/jira/browse/SPARK-8332
>>
>> This is a blocker for me to deploy a Spark dataframe based app on an
>> existing cluster, any input regarding how to create a proper classpath
>> would be great.
>>
>> Regards,
>>
>> Olivier.
>>
>>
>> Le mer. 17 juin 2015 à 11:37, Olivier Girardot <
>> o.girardot@lateral-thoughts.com> a écrit :
>>
>>> Hi everyone,
>>> After copying the hive-site.xml from a CDH5 cluster, I can't seem to
>>> connect to the hive metastore using spark-shell, here's a part of the stack
>>> trace I get :
>>>
>>> 15/06/17 04:41:57 ERROR TSaslTransport: SASL negotiation failure
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>> GSSException: No valid credentials provided (Mechanism level: Failed to
>>> find any Kerberos tgt)]
>>> at
>>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
>>> at
>>> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
>>> at
>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>> at
>>> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>>> at
>>> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>>> at
>>> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> The user has a non-expired ticket, I can execute hadoop fs -ls, all in
>>> all I should have access to this.
>>> I am stuck with this issue on Spark 1.4.0, did not try a version
>>> before...
>>>
>>> Any guess regarding what might be wrong ?
>>>
>>> Regards,
>>>
>>> Olivier.
>>>
>>
>

Mime
View raw message