spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-1384) spark-shell on yarn doesn't always work with secure hdfs
Date Tue, 01 Apr 2014 17:25:14 GMT
Thomas Graves created SPARK-1384:
------------------------------------

             Summary: spark-shell on yarn doesn't always work with secure hdfs
                 Key: SPARK-1384
                 URL: https://issues.apache.org/jira/browse/SPARK-1384
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 0.9.0, 0.9.1
            Reporter: Thomas Graves


 I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 rc3 release.  It
doesn't work with secure HDFS unless you 
export SPARK_YARN_MODE=true before starting the shell, or if you happen to do something immediately
with HDFS.  If you wait for the connection to the namenode to timeout it will fail. 

I think it was actually this way in the 0.9 release also so I thought I would send this and
get peoples feedback to see if you want it fixed? 

Another option would be to document that you have to export SPARK_YARN_MODE=true for the shell.
  The fix actually went in with the authentication changes I made in master but I never realized
that change needed to apply to 0.9. 

https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
See the SparkILoop diff.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message