spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liupengcheng (JIRA)" <>
Subject [jira] [Created] (SPARK-26877) Support user staging directory in yarn mode
Date Thu, 14 Feb 2019 07:58:00 GMT
liupengcheng created SPARK-26877:

             Summary: Support user staging directory in yarn mode
                 Key: SPARK-26877
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 2.4.0
            Reporter: liupengcheng

Currently, when running applications on yarn mode, the app staging directory of  is controlled
by `spark.yarn.stagingDir` config if specified, and this directory cannot separate different
users, somtimes, it's inconvenient for file and quota management for users.

For example, user may have different quota for their own app staging dir. and they may also
not want to use the home directory as the staging dir, because it might used mainly for private
user files or data. The quota may also be different between private data and the app staging

So I propose to add user sub directories under this app staging dir.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message