spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-1453) Improve the way Spark on Yarn waits for executors before starting
Date Wed, 09 Apr 2014 15:18:28 GMT
Thomas Graves created SPARK-1453:
------------------------------------

             Summary: Improve the way Spark on Yarn waits for executors before starting
                 Key: SPARK-1453
                 URL: https://issues.apache.org/jira/browse/SPARK-1453
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 1.0.0
            Reporter: Thomas Graves
            Assignee: Thomas Graves


Currently Spark on Yarn just delays a few seconds between when the spark context is initialized
and when it allows the job to start.  If you are on a busy hadoop cluster is might take longer
to get the number of executors. 

In the very least we could make this timeout a configurable value.  Its currently hardcoded
to 3 seconds.  
Better yet would be to allow user to give a minimum number of executors it wants to wait for,
but that looks much more complex. 




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message