spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-5222) YARN client and cluster modes have different app name behaviors
Date Tue, 13 Jan 2015 17:48:34 GMT

     [ https://issues.apache.org/jira/browse/SPARK-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Or updated SPARK-5222:
-----------------------------
    Description: 
The behavior is summarized in a table produced by [~WangTaoTheTonic] here: https://github.com/apache/spark/pull/3557

SPARK_YARN_APP_NAME is respected only in client mode but not in cluster mode. This results
in the strange behavior where the app name changes if the user runs the same application but
uses a different deploy mode from before. We should make sure the app name behavior is consistent
across deploy modes regardless of what variable or config is set.

Additionally, it should be noted that because "spark.app.name" is required of all applications,
the setting of "SPARK_YARN_APP_NAME" will not take effect unless we handle it preemptively
in Spark submit.

  was:
The behavior is summarized in a table produced by [~WangTaoTheTonic] here: https://github.com/apache/spark/pull/3557

SPARK_YARN_APP_NAME is respected only in client mode but not in cluster mode. This results
in the strange behavior where the app name changes if the user runs the same application but
uses a different deploy mode from before.

We should make sure the app name behavior is consistent across deploy modes regardless of
what variable or config is set.


> YARN client and cluster modes have different app name behaviors
> ---------------------------------------------------------------
>
>                 Key: SPARK-5222
>                 URL: https://issues.apache.org/jira/browse/SPARK-5222
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.0.0
>            Reporter: Andrew Or
>            Assignee: WangTaoTheTonic
>
> The behavior is summarized in a table produced by [~WangTaoTheTonic] here: https://github.com/apache/spark/pull/3557
> SPARK_YARN_APP_NAME is respected only in client mode but not in cluster mode. This results
in the strange behavior where the app name changes if the user runs the same application but
uses a different deploy mode from before. We should make sure the app name behavior is consistent
across deploy modes regardless of what variable or config is set.
> Additionally, it should be noted that because "spark.app.name" is required of all applications,
the setting of "SPARK_YARN_APP_NAME" will not take effect unless we handle it preemptively
in Spark submit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message