spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-9209) Using executor allocation, a executor is removed but it exists in ExecutorsPage of the web ui
Date Wed, 13 Feb 2019 21:27:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Vanzin resolved SPARK-9209.
-----------------------------------
    Resolution: Not A Problem

Dead executors are now explicitly kept by Spark, so this is not an issue anymore (it's become
a feature!).

> Using executor allocation, a executor is removed but it exists in ExecutorsPage of the
web ui 
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-9209
>                 URL: https://issues.apache.org/jira/browse/SPARK-9209
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.4.1
>            Reporter: KaiXinXIaoLei
>            Priority: Minor
>         Attachments: A Executor exists in web.png, executor is removed.png
>
>
> I set "spark.dynamicAllocation.enabled = true”, and  run a big job. In driver, a executor
is asked to remove, and it's remove successfully, and the process of this executor is not
exist. But it exists in ExecutorsPage of the web ui.
> The log in driver :
> 2015-07-17 11:48:14,543 | INFO  | [sparkDriver-akka.actor.default-dispatcher-3] | Removing
block manager BlockManagerId(264, 172.1.1.8, 23811) 
> 2015-07-17 11:48:14,543 | INFO  | [dag-scheduler-event-loop] | Removed 264 successfully
in removeExecutor 
> 2015-07-17 11:48:21,226 | INFO  | [sparkDriver-akka.actor.default-dispatcher-3] | Registering
block manager 172.1.1.8:23811 with 10.4 GB RAM, BlockManagerId(264, 172.1.1.8, 23811) 
> 2015-07-17 11:48:21,228 | INFO  | [sparkDriver-akka.actor.default-dispatcher-3] | Added
broadcast_781_piece0 in memory on 172.1.1.8:23811 (size: 38.6 KB, free: 10.4 GB)  
> 2015-07-17 11:48:35,277 | ERROR | [sparkDriver-akka.actor.default-dispatcher-16] | Lost
executor 264 on datasight-195: remote Rpc client disassociated 
> 2015-07-17 11:48:35,277 | WARN  | [sparkDriver-akka.actor.default-dispatcher-4] | Association
with remote system [akka.tcp://sparkExecutor@datasight-195:23929] has failed, address is now
gated for [5000] ms. Reason is: [Disassociated].
> 2015-07-17 11:48:35,277 | INFO  | [sparkDriver-akka.actor.default-dispatcher-16] | Re-queueing
tasks for 264 from TaskSet 415.0 
> 2015-07-17 11:48:35,804 | INFO  | [SparkListenerBus] | Existing executor 264 has been
removed (new total is 10)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message