spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-24992) spark should randomize yarn local dir selection
Date Thu, 02 Aug 2018 02:06:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-24992:
------------------------------------

    Assignee:     (was: Apache Spark)

> spark should randomize yarn local dir selection
> -----------------------------------------------
>
>                 Key: SPARK-24992
>                 URL: https://issues.apache.org/jira/browse/SPARK-24992
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.3.1
>            Reporter: Hieu Tri Huynh
>            Priority: Minor
>
> Utils.getLocalDir is used to get path of a temporary directory. However, it always
returns the the same directory, which is the first element in the array _localRootDirs_. When
running on YARN, this might causes the case that we always write to one disk, which makes
it busy while other disks are free. We should randomize the selection to spread out the loads. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message