spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-1384) spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs
Date Wed, 02 Apr 2014 13:41:15 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Graves reassigned SPARK-1384:
------------------------------------

    Assignee: Thomas Graves

> spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-1384
>                 URL: https://issues.apache.org/jira/browse/SPARK-1384
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 0.9.0, 0.9.1
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>
>  I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 rc3 release.
 It doesn't work with secure HDFS unless you 
> export SPARK_YARN_MODE=true before starting the shell, or if you happen to do something
immediately with HDFS.  If you wait for the connection to the namenode to timeout it will
fail. 
>  
> The fix actually went in to master branch  with the authentication changes I made in
master but I never realized that change needed to apply to 0.9. 
> https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
> See the SparkILoop diff.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message