spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-6469) Local directories configured for YARN are not used in yarn-client mode
Date Mon, 23 Mar 2015 15:53:11 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376092#comment-14376092
] 

Thomas Graves commented on SPARK-6469:
--------------------------------------

Yeah so you should be able to set the spark.local.dir config to change that directory in yarn
client mode for the driver.  Executors will still use the yarn approved directories. 

We should change this jira to clarify documentation then. 

> Local directories configured for YARN are not used in yarn-client mode
> ----------------------------------------------------------------------
>
>                 Key: SPARK-6469
>                 URL: https://issues.apache.org/jira/browse/SPARK-6469
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Christophe PR√ČAUD
>            Priority: Minor
>         Attachments: TestYarnVars.scala
>
>
> According to the [Spark YARN doc page|http://spark.apache.org/docs/latest/running-on-yarn.html#important-notes],
Spark executors will use the local directories configured for YARN, not spark.local.dir which
should be ignored.
> If this works correctly in yarn-cluster mode, I've found out that it is not the case
in yarn-client mode.
> The problem seems to originate in the method [isRunningInYarnContainer|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L686].
> Indeed, I've checked with a simple application that the {{CONTAINER_ID}} environment
variable is correctly set in yarn-cluster mode (to something like {{container_1426666761810_0151_01_000001}},
but not in yarn-client mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message