spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Mahadevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21029) All StreamingQuery should be stopped when the SparkSession is stopped
Date Wed, 27 Feb 2019 21:05:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779765#comment-16779765
] 

Arun Mahadevan commented on SPARK-21029:
----------------------------------------

I encountered this issue and propose an alternative here https://github.com/apache/spark/pull/23912

To keep it simple, this just stops the streaming queries when the spark session is stopped.
It does not handle the spark context being directly stopped, since that requires more changes
to add a callback hook that the spark context can invoke and the session can leverage. Using
the shutdown hook like proposed previously does not look clean either.

> All StreamingQuery should be stopped when the SparkSession is stopped
> ---------------------------------------------------------------------
>
>                 Key: SPARK-21029
>                 URL: https://issues.apache.org/jira/browse/SPARK-21029
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.2.0, 2.3.0
>            Reporter: Felix Cheung
>            Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message