spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-26877) Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified
Date Thu, 14 Feb 2019 08:13:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-26877:
------------------------------------

    Assignee: Apache Spark

> Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-26877
>                 URL: https://issues.apache.org/jira/browse/SPARK-26877
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 2.4.0
>            Reporter: liupengcheng
>            Assignee: Apache Spark
>            Priority: Minor
>
> Currently, when running applications on yarn mode, the app staging directory of  is
controlled by `spark.yarn.stagingDir` config if specified, and this directory cannot separate
different users, somtimes, it's inconvenient for file and quota management for users.
> For example, user may have different quota for their own app staging dir. and they may
also not want to use the home directory as the staging dir, because it might used mainly for
private user files or data. The quota may also be different between private data and the app
staging dir.
> So I propose to add user sub directories under this app staging dir.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message