spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Burak Yavuz (Jira)" <>
Subject [jira] [Created] (SPARK-30143) StreamingQuery.stop() should not block indefinitely
Date Thu, 05 Dec 2019 22:34:00 GMT
Burak Yavuz created SPARK-30143:

             Summary: StreamingQuery.stop() should not block indefinitely
                 Key: SPARK-30143
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.4.4
            Reporter: Burak Yavuz

The stop() method on a Streaming Query awaits the termination of the stream execution thread.
However, the stream execution thread may block forever depending on the streaming source implementation
(like in Kafka, which runs UninterruptibleThreads).

This causes control flow applications to hang indefinitely as well. We'd like to introduce
a timeout to stop the execution thread, so that the control flow thread can decide to do an
action if a timeout is hit. 

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message