spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hieu Tri Huynh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-24992) spark should randomize yarn local dir selection
Date Wed, 01 Aug 2018 22:14:00 GMT
Hieu Tri Huynh created SPARK-24992:
--------------------------------------

             Summary: spark should randomize yarn local dir selection
                 Key: SPARK-24992
                 URL: https://issues.apache.org/jira/browse/SPARK-24992
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.3.1
            Reporter: Hieu Tri Huynh


Utils.getLocalDir is used to get path of a temporary directory. However, it always returns
the the same directory, which is the first element in the array _localRootDirs_. When running
on YARN, this might causes the case that we always write to one disk, which makes it busy
while other disks are free. We should randomize the selection to spread out the loads. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message