spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aneela Saleem <ane...@platalytics.com>
Subject Re: submitting spark job with kerberized Hadoop issue
Date Tue, 16 Aug 2016 13:12:25 GMT
Thanks Steve,

I went through this but still not able to fix the issue

On Mon, Aug 15, 2016 at 2:01 AM, Steve Loughran <stevel@hortonworks.com>
wrote:

> Hi,
>
> Just came across this while going through all emails I'd left unread over
> my vacation.
>
> did you manage to fix this?
>
> 1. There's some notes I've taken on this topic:
> https://www.gitbook.com/book/steveloughran/kerberos_and_hadoop/details
>
>  -look at "Error messages to fear" to see if this one has surfaced;
> otherwise look at "low level secrets" to see how to start debugging things
>
>
> On 5 Aug 2016, at 14:54, Aneela Saleem <aneela@platalytics.com> wrote:
>
> Hi all,
>
> I'm trying to connect to Kerberized Hadoop cluster using spark job. I have
> kinit'd from command line. When i run the following job i.e.,
>
> *./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal
> spark/hadoop-master@platalyticsrealm --class
> com.platalytics.example.spark.App --master spark://hadoop-master:7077
> /home/vm6/project-1-jar-with-dependencies.jar
> hdfs://hadoop-master:8020/text*
>
> I get the error:
>
> Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException:
> Client cannot authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1628)
>
> Following are the contents of *spark-defaults.conf* file:
>
> spark.master                     spark://hadoop-master:7077
> spark.eventLog.enabled           true
> spark.eventLog.dir               hdfs://hadoop-master:8020/spark/logs
> spark.serializer                 org.apache.spark.serializer.
> KryoSerializer
> spark.yarn.access.namenodes hdfs://hadoop-master:8020/
> spark.yarn.security.tokens.hbase.enabled true
> spark.yarn.security.tokens.hive.enabled true
> spark.yarn.principal yarn/hadoop-master@platalyticsrealm
> spark.yarn.keytab /etc/hadoop/conf/yarn.keytab
>
>
> Also i have added following in *spark-env.sh* file:
>
> HOSTNAME=`hostname -f`
> export SPARK_HISTORY_OPTS="-Dspark.history.kerberos.enabled=true
> -Dspark.history.kerberos.principal=spark/${HOSTNAME}@platalyticsrealm
> -Dspark.history.kerberos.keytab=/etc/hadoop/conf/spark.keytab"
>
>
> Please guide me, how to trace the issue?
>
> Thanks
>
>
>

Mime
View raw message