spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-30391) ExecutorAllocationManager requestTotalExecutors in removeExecutors may not be needed
Date Mon, 30 Dec 2019 16:26:01 GMT
Thomas Graves created SPARK-30391:
-------------------------------------

             Summary: ExecutorAllocationManager requestTotalExecutors in removeExecutors may
not be needed
                 Key: SPARK-30391
                 URL: https://issues.apache.org/jira/browse/SPARK-30391
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.4, 3.0.0
            Reporter: Thomas Graves


in the ExecutorAllocationManager.removeExecutors

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L459]

 

there is a call to requestTotalExecutors that I don't think is needed anymore because SPARK-23365
changed the killExecutors call to not adjust the target number. We should investigate if we
can remove it.

{color:#808080}
{color}{color:#808080}
{color}{color:#808080}// [SPARK-21834] killExecutors api reduces the target number of executors.
{color}{color:#808080}// So we need to update the target with desired value.
{color}client.requestTotalExecutors({color:#660e7a}numLocalityAwareTasksPerResourceProfileId{color}.toMap,
 {color:#660e7a}rpIdToHostToLocalTaskCount{color}, {color:#660e7a}numExecutorsTargetPerResourceProfileId{color}.toMap)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message