spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <>
Subject [jira] [Created] (SPARK-1384) spark-shell on yarn doesn't always work with secure hdfs
Date Tue, 01 Apr 2014 17:25:14 GMT
Thomas Graves created SPARK-1384:

             Summary: spark-shell on yarn doesn't always work with secure hdfs
                 Key: SPARK-1384
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 0.9.0, 0.9.1
            Reporter: Thomas Graves

 I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 rc3 release.  It
doesn't work with secure HDFS unless you 
export SPARK_YARN_MODE=true before starting the shell, or if you happen to do something immediately
with HDFS.  If you wait for the connection to the namenode to timeout it will fail. 

I think it was actually this way in the 0.9 release also so I thought I would send this and
get peoples feedback to see if you want it fixed? 

Another option would be to document that you have to export SPARK_YARN_MODE=true for the shell.
  The fix actually went in with the authentication changes I made in master but I never realized
that change needed to apply to 0.9.
See the SparkILoop diff.

This message was sent by Atlassian JIRA

View raw message