spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "L. C. Hsieh (Jira)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude
Date Tue, 02 Feb 2021 05:56:01 GMT

     [ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

L. C. Hsieh reassigned SPARK-34295:
-----------------------------------

    Assignee: L. C. Hsieh

> Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude
> ------------------------------------------------------------------------
>
>                 Key: SPARK-34295
>                 URL: https://issues.apache.org/jira/browse/SPARK-34295
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 3.2.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>
> MapReduce jobs can instruct YARN to skip renewal of tokens obtained from certain hosts
by specifying the hosts with configuration mapreduce.job.hdfs-servers.token-renewal.exclude=<host1>,<host2>,..,<hostN>.
> But seems Spark lacks of similar option. So the job submission fails if YARN fails to
renew DelegationToken for any of the remote HDFS cluster.  The failure in DT renewal can happen
due to many reason like Remote HDFS does not trust Kerberos identity of YARN etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message