spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Casey <gerardhughca...@gmail.com>
Subject Re: Kerberos and YARN - functions in spark-shell and spark submit local but not cluster mode
Date Mon, 05 Dec 2016 18:44:03 GMT
Thanks Marcelo,

My understanding from a few pointers is that this may be due to insufficient read permissions
to the key tab or a corrupt key tab. I have checked the read permissions and they are ok.
I can see that it is initially configuring correctly:

       INFO security.UserGroupInformation: Login successful for user user@login_node using
keytab file /path/to/keytab

I’ve added the full trace below. 

Gerry

Full trace:

Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
16/12/05 18:23:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
16/12/05 18:23:27 INFO security.UserGroupInformation: Login successful for user me@login_nodeusing
keytab file /path/to/keytab
16/12/05 18:23:27 INFO yarn.Client: Attempting to login to the Kerberos using principal: me@login_node
and keytab: /path/to/keytab
16/12/05 18:23:28 INFO impl.TimelineClientImpl: Timeline service address: http://login_node1.xcat.cluster:8188/ws/v1/timeline/
16/12/05 18:23:28 INFO client.RMProxy: Connecting to ResourceManager at login_node1.xcat.cluster/
16/12/05 18:23:28 INFO client.AHSProxy: Connecting to Application History server at login_node1.xcat.cluster/
16/12/05 18:23:28 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature
cannot be used because libhadoop cannot be loaded.
16/12/05 18:23:28 INFO yarn.Client: Requesting a new application from cluster with 32 NodeManagers
16/12/05 18:23:28 INFO yarn.Client: Verifying our application has not requested more than
the maximum memory capability of the cluster (15360 MB per container)
16/12/05 18:23:28 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including
384 MB overhead
16/12/05 18:23:28 INFO yarn.Client: Setting up container launch context for our AM
16/12/05 18:23:28 INFO yarn.Client: Setting up the launch environment for our AM container
16/12/05 18:23:28 INFO yarn.Client: Using the spark assembly jar on HDFS because you are using
HDP, defaultSparkAssembly:hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar
16/12/05 18:23:28 INFO yarn.Client: Credentials file set to:
16/12/05 18:23:28 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_
16/12/05 18:23:28 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1856 for me on
16/12/05 18:23:28 INFO yarn.Client: Renewal Interval set to 86400009
16/12/05 18:23:28 INFO yarn.Client: Preparing resources for our AM container
16/12/05 18:23:28 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_
16/12/05 18:23:28 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1857 for me on

16/12/05 18:23:29 INFO yarn.YarnSparkHadoopUtil: HBase class not found java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.HBaseConfiguration
16/12/05 18:23:29 INFO yarn.Client: To enable the AM to login from keytab, credentials are
being copied over to the AM via the YARN Secure Distributed Cache.
16/12/05 18:23:29 INFO yarn.Client: Uploading resource file:/path/to/keytab -> hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/keytab
16/12/05 18:23:29 INFO yarn.Client: Using the spark assembly jar on HDFS because you are using
HDP, defaultSparkAssembly:hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar
16/12/05 18:23:29 INFO yarn.Client: Source and destination file systems are the same. Not
copying hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar
16/12/05 18:23:29 INFO yarn.Client: Uploading resource file:/home/me/Aoife/spark-abm/target/scala-2.10/graphx_sp_2.10-1.0.jar
-> hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/graphx_sp_2.10-1.0.jar
16/12/05 18:23:29 INFO yarn.Client: Uploading resource file:/tmp/spark-2e566133-d50a-4904-920e-ab5cec07c644/__spark_conf__6538744395325375994.zip
-> hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/__spark_conf__6538744395325375994.zip
16/12/05 18:23:29 INFO spark.SecurityManager: Changing view acls to: me
16/12/05 18:23:29 INFO spark.SecurityManager: Changing modify acls to: me
16/12/05 18:23:29 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(me); users with modify permissions: Set(me)
16/12/05 18:23:29 INFO yarn.Client: Submitting application 41 to ResourceManager
16/12/05 18:23:30 INFO impl.YarnClientImpl: Submitted application application_1479877553404_0041
16/12/05 18:23:31 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:31 INFO yarn.Client:
	 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1480962209903
	 final status: UNDEFINED
	 tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
	 user: me
16/12/05 18:23:32 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:33 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:34 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:35 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:36 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:37 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:38 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:39 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:40 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:40 INFO yarn.Client:
	 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
	 diagnostics: N/A
	 ApplicationMaster host: 
	 ApplicationMaster RPC port: 0
	 queue: default
	 start time: 1480962209903
	 final status: UNDEFINED
	 tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
	 user: me
16/12/05 18:23:41 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:42 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:43 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:44 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:45 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:46 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:47 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:48 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:49 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:50 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:51 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:52 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:53 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:54 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:23:55 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:55 INFO yarn.Client:
	 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1480962209903
	 final status: UNDEFINED
	 tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
	 user: me
16/12/05 18:23:56 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:57 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:58 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:23:59 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:24:00 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:24:01 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:24:02 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:24:03 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:24:04 INFO yarn.Client: Application report for application_1479877553404_0041
(state: ACCEPTED)
16/12/05 18:24:05 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:05 INFO yarn.Client:
	 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
	 diagnostics: N/A
	 ApplicationMaster host: 
	 ApplicationMaster RPC port: 0
	 queue: default
	 start time: 1480962209903
	 final status: UNDEFINED
	 tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
	 user: me
16/12/05 18:24:06 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:07 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:08 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:09 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:10 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:11 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:12 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:13 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:14 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:15 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:16 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:17 INFO yarn.Client: Application report for application_1479877553404_0041
(state: RUNNING)
16/12/05 18:24:18 INFO yarn.Client: Application report for application_1479877553404_0041
(state: FINISHED)
16/12/05 18:24:18 INFO yarn.Client:
	 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
	 diagnostics: User class threw exception: org.apache.hadoop.security.AccessControlException:
Authentication required
	 ApplicationMaster host: 
	 ApplicationMaster RPC port: 0
	 queue: default
	 start time: 1480962209903
	 final status: FAILED
	 tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
	 user: me
Exception in thread "main" org.apache.spark.SparkException: Application application_1479877553404_0041
finished with failed status
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1122)
	at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1169)
	at org.apache.spark.deploy.yarn.Client.main(Client.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/12/05 18:24:18 INFO util.ShutdownHookManager: Shutdown hook called
16/12/05 18:24:18 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2e566133-d50a-4904-920e-ab5cec07c644



On Mon, Dec 5, 2016 at 10:30 AM, Gerard Casey <gerardhughcasey@gmail.com> wrote:

> 
>> On 5 Dec 2016, at 19:26, Marcelo Vanzin <vanzin@cloudera.com> wrote:
>> 
>> There's generally an exception in these cases, and you haven't posted
>> it, so it's hard to tell you what's wrong. The most probable cause,
>> without the extra information the exception provides, is that you're
>> using the wrong Hadoop configuration when submitting the job to YARN.
>> 
>> On Mon, Dec 5, 2016 at 4:35 AM, Gerard Casey <gerardhughcasey@gmail.com> wrote:
>>> Hello all,
>>> 
>>> I am using Spark with Kerberos authentication.
>>> 
>>> I can run my code using `spark-shell` fine and I can also use `spark-submit`
>>> in local mode (e.g. —master local[16]). Both function as expected.
>>> 
>>> local mode -
>>> 
>>> spark-submit --class "graphx_sp" --master local[16] --driver-memory 20G
>>> target/scala-2.10/graphx_sp_2.10-1.0.jar
>>> 
>>> I am now progressing to run in cluster mode using YARN.
>>> 
>>> cluster mode with YARN -
>>> 
>>> spark-submit --class "graphx_sp" --master yarn --deploy-mode cluster
>>> --executor-memory 13G --total-executor-cores 32
>>> target/scala-2.10/graphx_sp_2.10-1.0.jar
>>> 
>>> However, this returns:
>>> 
>>> diagnostics: User class threw exception:
>>> org.apache.hadoop.security.AccessControlException: Authentication required
>>> 
>>> Before I run using spark-shell or on local mode in spark-submit I do the
>>> following kerberos setup:
>>> 
>>> kinit -k -t ~/keytab -r 7d `whoami`
>>> 
>>> Clearly, this setup is not extending to the YARN setup. How do I fix the
>>> Kerberos issue with YARN in cluster mode? Is this something which must be in
>>> my /src/main/scala/graphx_sp.scala file?
>>> 
>>> Many thanks
>>> 
>>> Geroid
>> 
>> 
>> 
>> --
>> Marcelo
> 



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message