spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saisai Shao (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-18840) HDFSCredentialProvider throws exception in non-HDFS security environment
Date Tue, 13 Dec 2016 09:30:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744370#comment-15744370
] 

Saisai Shao edited comment on SPARK-18840 at 12/13/16 9:30 AM:
---------------------------------------------------------------

This problem also existed in branch 1.6, but the fix is a little different compared to master.


was (Author: jerryshao):
This problem also existed in branch 1.6, but the fix is a little complicated compared to master.

> HDFSCredentialProvider throws exception in non-HDFS security environment
> ------------------------------------------------------------------------
>
>                 Key: SPARK-18840
>                 URL: https://issues.apache.org/jira/browse/SPARK-18840
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.6.3, 2.1.0
>            Reporter: Saisai Shao
>            Priority: Minor
>
> Current in {{HDFSCredentialProvider}}, the code logic assumes HDFS delegation token should
be existed, this is ok for HDFS environment, but for some cloud environment like Azure, HDFS
is not required, so it will throw exception:
> {code}
> java.util.NoSuchElementException: head of empty list
>         at scala.collection.immutable.Nil$.head(List.scala:337)
>         at scala.collection.immutable.Nil$.head(List.scala:334)
>         at org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval(Client.scala:627)
> {code}
> We should also consider this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message