spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christophe PRÉAUD (JIRA) <>
Subject [jira] [Updated] (SPARK-6469) Local directories configured for YARN are not used in yarn-client mode
Date Mon, 23 Mar 2015 14:15:13 GMT


Christophe PRÉAUD updated SPARK-6469:
    Attachment: TestYarnVars.scala

Attached a simple application to check the value of the {{CONTAINER_ID}} environment variable.

* Check in yarn-cluster mode
/opt/spark/bin/spark-submit --master yarn-cluster --class TestYarnVars --queue spark-batch
testyarnvars_2.10-1.0.jar 2>/dev/null
(the stdout of the application on the YARN wen ui reads: {{CONTAINER_ID: container_1426666761810_0151_01_000001}}

* Check in yarn-client mode:
/opt/spark/bin/spark-submit --master yarn-client --class TestYarnVars --queue spark-batch
testyarnvars_2.10-1.0.jar 2>/dev/null

> Local directories configured for YARN are not used in yarn-client mode
> ----------------------------------------------------------------------
>                 Key: SPARK-6469
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Christophe PRÉAUD
>            Priority: Minor
>         Attachments: TestYarnVars.scala
> According to the [Spark YARN doc page|],
Spark executors will use the local directories configured for YARN, not spark.local.dir which
should be ignored.
> If this works correctly in yarn-cluster mode, I've found out that it is not the case
in yarn-client mode.
> The problem seems to originate in the method [isRunningInYarnContainer|].
> Indeed, I've checked with a simple application that the {{CONTAINER_ID}} environment
variable is correctly set in yarn-cluster mode (to something like {{container_1426666761810_0151_01_000001}},
but not in yarn-client mode.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message