spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "haiyangyu (Jira)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-30297) Executor heartbeat expired cause app hung up forever
Date Wed, 18 Dec 2019 13:35:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

haiyangyu updated SPARK-30297:
------------------------------
    Description: 
h3. *Backgroud*

The driver can't sense this executor was lost through the network connection disconnection
If an executor was lost in the network and it have not responsed rst and close packet to driver,
so driver can only sense this executor dead through heartbeat expired.
h3. *Problems*

Heartbeat expiration processing flow as follows:
 # Executor heartbeat expired as above.
 # HeartbeatReceiver will call scheduler executor lost to rescheduler the tasks on this executor.
 # HeartbeatReceiver kill the executor.

The tasks on the dead executor have a chance to rescheduled on this dead executor again if
the task rescheduler before the executor has't remove from executorBackend, it will send launch
task to this executor again, the executor will not response and the driver can't sense through
heartbeat beause the executor has lost in network. This cause those tasks rescheduled on this
lost executor can't finish forever, and the app will hung up here forever.

  was:
h3. *Backgroud*

The driver can't sense this executor was lost through the network connection disconnection
If an executor was lost in the network and it have not responsed rst and close packet to driver,
so driver can only sense this executor dead through heartbeat expired.
h3. *Scene*
 # Executor heartbeat expired as above.
 # HeartbeatReceiver will call scheduler executor lost to rescheduler the tasks on this executor.
 # HeartbeatReceiver kill the executor.

The task on the dead executor will be rescheduled on this dead executor again if the task
rescheduler before the executor has't remove from executorBackend, it will send launch task
to this executor again, the executor will not response and the driver can't sense through
heartbeat beause the executor has lost in network.


> Executor heartbeat expired cause app hung up forever
> ----------------------------------------------------
>
>                 Key: SPARK-30297
>                 URL: https://issues.apache.org/jira/browse/SPARK-30297
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.4.0, 2.4.4
>            Reporter: haiyangyu
>            Priority: Major
>
> h3. *Backgroud*
> The driver can't sense this executor was lost through the network connection disconnection
If an executor was lost in the network and it have not responsed rst and close packet to driver,
so driver can only sense this executor dead through heartbeat expired.
> h3. *Problems*
> Heartbeat expiration processing flow as follows:
>  # Executor heartbeat expired as above.
>  # HeartbeatReceiver will call scheduler executor lost to rescheduler the tasks on this
executor.
>  # HeartbeatReceiver kill the executor.
> The tasks on the dead executor have a chance to rescheduled on this dead executor again
if the task rescheduler before the executor has't remove from executorBackend, it will send
launch task to this executor again, the executor will not response and the driver can't sense
through heartbeat beause the executor has lost in network. This cause those tasks rescheduled
on this lost executor can't finish forever, and the app will hung up here forever.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message