spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-11662) Call startExecutorDelegationTokenRenewer() ahead of client app submission
Date Tue, 12 Feb 2019 20:45:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-11662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Vanzin resolved SPARK-11662.
------------------------------------
    Resolution: Cannot Reproduce

Pretty sure this works these days.

> Call startExecutorDelegationTokenRenewer() ahead of client app submission
> -------------------------------------------------------------------------
>
>                 Key: SPARK-11662
>                 URL: https://issues.apache.org/jira/browse/SPARK-11662
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>            Reporter: Ted Yu
>            Priority: Major
>
> As reported in the thread 'Creating new Spark context when running in Secure YARN fails',
IOException may be thrown when SparkContext is stopped and started again working with secure
YARN cluster:
> {code}
> 15/11/11 10:19:53 ERROR spark.SparkContext: Error initializing SparkContext.
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:6638)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:563)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:987)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1476)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1407)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>         at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:933)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy13.getDelegationToken(Unknown Source)
>         at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1044)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1543)
>         at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:530)
>         at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2228)
>         at
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:126)
>         at
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:123)
>         at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
>         at
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokensForNamenodes(YarnSparkHadoopUtil.scala:123)
>         at
> org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval(Client.scala:495)
>         at
> org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:528)
>         at
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:628)
>         at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
>         at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
>         at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
> {code}
> One fix is to call startExecutorDelegationTokenRenewer(conf) ahead of client app submission.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message