spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Casey <gerardhughca...@gmail.com>
Subject Re: Kerberos and YARN - functions in spark-shell and spark submit local but not cluster mode
Date Mon, 05 Dec 2016 12:49:59 GMT
Edit. From here <http://www.cloudera.com/documentation/enterprise/latest/topics/sg_spark_auth.html>
I read that you can pass a `key tab` option to spark-submit. I thus tried

spark-submit --class "graphx_sp" --master yarn  --keytab /path/to/keytab  --deploy-mode cluster
--executor-memory 13G --total-executor-cores 32 target/scala-2.10/graphx_sp_2.10-1.0.jar

However, the error persists

Any ideas?

Thanks

Geroid 

> On 5 Dec 2016, at 13:35, Gerard Casey <gerardhughcasey@gmail.com> wrote:
> 
> Hello all,
> 
> I am using Spark with Kerberos authentication.
> 
> I can run my code using `spark-shell` fine and I can also use `spark-submit` in local
mode (e.g. —master local[16]). Both function as expected.
> 
> local mode -
> 
> 	spark-submit --class "graphx_sp" --master local[16] --driver-memory 20G target/scala-2.10/graphx_sp_2.10-1.0.jar
> 
> I am now progressing to run in cluster mode using YARN.
> 
> cluster mode with YARN - 
> 
> 	spark-submit --class "graphx_sp" --master yarn --deploy-mode cluster --executor-memory
13G --total-executor-cores 32 target/scala-2.10/graphx_sp_2.10-1.0.jar
> 
> However, this returns:
> 
> 	diagnostics: User class threw exception: org.apache.hadoop.security.AccessControlException:
Authentication required
> 
> Before I run using spark-shell or on local mode in spark-submit I do the following kerberos
setup:
> 
> 	kinit -k -t ~/keytab -r 7d `whoami`
> 
> Clearly, this setup is not extending to the YARN setup. How do I fix the Kerberos issue
with YARN in cluster mode? Is this something which must be in my /src/main/scala/graphx_sp.scala
file? 
> 
> Many thanks
> 
> Geroid 


Mime
View raw message