spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Girardot <o.girar...@lateral-thoughts.com>
Subject Re: Spark Shell Hive Context and Kerberos ticket
Date Wed, 17 Jun 2015 19:08:37 GMT
Ok what was wrong was that the spark-env did not contain the
HADOOP_CONF_DIR properly set to /etc/hadoop/conf/
With that fixed, this issue is gone, but I can't seem to get Spark SQL
1.4.0 with Hive working on CDH 5.3 or 5.4 :
Using this command line :
IPYTHON=1 /.../spark-1.4.0-bin-hadoop2.4/bin/pyspark  --master yarn-client
--driver-class-path `hadoop classpath`

I end up with this issue :
: java.lang.NoSuchMethodError:
com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;
at
com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<init>(ScalaNumberDeserializersModule.scala:49)
at
com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<clinit>(ScalaNumberDeserializersModule.scala)
at
com.fasterxml.jackson.module.scala.deser.ScalaNumberDeserializersModule$class.$init$(ScalaNumberDeserializersModule.scala:61)
at
com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:19)
at
com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:35)
at
com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
at
org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:78)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:118)
at
org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:125)
at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1255)
at org.apache.spark.sql.DataFrame.head(DataFrame.scala:1189)
at org.apache.spark.sql.DataFrame.take(DataFrame.scala:1248)
at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:745)

This seems to be related to this Jira Issue :
https://issues.apache.org/jira/browse/SPARK-8332

This is a blocker for me to deploy a Spark dataframe based app on an
existing cluster, any input regarding how to create a proper classpath
would be great.

Regards,

Olivier.


Le mer. 17 juin 2015 à 11:37, Olivier Girardot <
o.girardot@lateral-thoughts.com> a écrit :

> Hi everyone,
> After copying the hive-site.xml from a CDH5 cluster, I can't seem to
> connect to the hive metastore using spark-shell, here's a part of the stack
> trace I get :
>
> 15/06/17 04:41:57 ERROR TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> at java.security.AccessController.doPrivileged(Native Method)
>
> The user has a non-expired ticket, I can execute hadoop fs -ls, all in all
> I should have access to this.
> I am stuck with this issue on Spark 1.4.0, did not try a version before...
>
> Any guess regarding what might be wrong ?
>
> Regards,
>
> Olivier.
>

Mime
View raw message