spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wojciech Indyk <wojciechin...@gmail.com>
Subject Delegation of Kerberos tokens
Date Fri, 08 Apr 2016 09:01:30 GMT
Hello!
 TL;DR Could you explain how (and which) Kerberos tokens should be
delegated from driver to workers? Does it depend on spark mode?

I have a Hadoop cluster HDP 2.3 with Kerberos. I use spark-sql (1.6.1
compiled with hadoop 2.7.1 and hive 1.2.1) on yarn-cluster mode to
query my hive tables.
1. When I query hive table stored in HDFS everything is fine. (assume
there is no problem with my app, config and credentials setup)
2. When I try to query external table of HBase (defined in Hive using
HBaseHandler) I have a permissions problem on RPC call from
Spark-workers to HBase region server. (there is no problem to connect
HBaseMaster from driver, Zookeepers from both driver and workers)
3. When I query the HBase table by hive (beeswax) everything is ok.
(assume there is no problem with HBaseHandler)

After some time of debugging (and write some additional logging) I see
the driver has (and delegates) only:
16/03/31 15:03:52 DEBUG YarnSparkHadoopUtil: token for:
16/03/31 15:03:52 DEBUG YarnSparkHadoopUtil: token for: 172.xx.xx102:8188
16/03/31 15:03:52 DEBUG YarnSparkHadoopUtil: token for: ha-hdfs:dataocean
Which means there are only credentials for YARN and HDFS. I am curious
is it proper behavior? I see another user has similar doubt:
https://issues.apache.org/jira/browse/SPARK-12279?focusedCommentId=15067020&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15067020

Could you explain how (and which) Kerberos tokens should be delegated
from driver to workers? Does it depend on spark mode? As I saw in the
code the method obtainTokenForHBase is calling when yarn-client mode
is on, but not for yarn-cluster. Am I right? Is it ok?

--
Kind regards/ Pozdrawiam,
Wojciech Indyk
http://datacentric.pl

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message