spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1384) spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs
Date Tue, 01 Apr 2014 17:25:15 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Graves updated SPARK-1384:
---------------------------------

    Summary: spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs
 (was: spark-shell on yarn doesn't always work with secure hdfs)

> spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-1384
>                 URL: https://issues.apache.org/jira/browse/SPARK-1384
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 0.9.0, 0.9.1
>            Reporter: Thomas Graves
>
>  I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 rc3 release.
 It doesn't work with secure HDFS unless you 
> export SPARK_YARN_MODE=true before starting the shell, or if you happen to do something
immediately with HDFS.  If you wait for the connection to the namenode to timeout it will
fail. 
> I think it was actually this way in the 0.9 release also so I thought I would send this
and get peoples feedback to see if you want it fixed? 
> Another option would be to document that you have to export SPARK_YARN_MODE=true for
the shell.   The fix actually went in with the authentication changes I made in master but
I never realized that change needed to apply to 0.9. 
> https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
> See the SparkILoop diff.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message