spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesper Lundgren (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-4986) Graceful shutdown for Spark Streaming does not work in Standalone cluster mode
Date Mon, 29 Dec 2014 07:34:13 GMT

     [ https://issues.apache.org/jira/browse/SPARK-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jesper Lundgren updated SPARK-4986:
-----------------------------------
    Description: 
When using the graceful stop API of Spark Streaming in Spark Standalone cluster the stop signal
never reaches the receivers. I have tested this with Spark 1.2 and Kafka receivers. 

ReceiverTracker will send StopReceiver message to ReceiverSupervisorImpl.
In local mode ReceiverSupervisorImpl receives this message but in Standalone cluster mode
the message seems to be lost.

(I have modified the code to send my own string message as a stop signal from ReceiverTracker
to ReceiverSupervisorImpl and it works as a workaround.)

  was:
When using the graceful stop API of Spark Streaming in Spark Standalone cluster the stop signal
never reaches the receivers. I have tested this with Spark 1.2 and Kafka receivers. 

ReceiverTracker will send StopReceiver message to ReceiverSupervisorImpl.
In local mode ReceiverSupervisorImpl receives this message but in Standalone cluster mode
the message seems to be lost.

Sent my own string message as a stop signal from ReceiverTracker to ReceiverSupervisorImpl
and it works as a workaround.


> Graceful shutdown for Spark Streaming does not work in Standalone cluster mode
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-4986
>                 URL: https://issues.apache.org/jira/browse/SPARK-4986
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.2.0
>            Reporter: Jesper Lundgren
>
> When using the graceful stop API of Spark Streaming in Spark Standalone cluster the stop
signal never reaches the receivers. I have tested this with Spark 1.2 and Kafka receivers.

> ReceiverTracker will send StopReceiver message to ReceiverSupervisorImpl.
> In local mode ReceiverSupervisorImpl receives this message but in Standalone cluster
mode the message seems to be lost.
> (I have modified the code to send my own string message as a stop signal from ReceiverTracker
to ReceiverSupervisorImpl and it works as a workaround.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message