spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongjoon Hyun <>
Subject Re: [DISCUSS] Disable streaming query with possible correctness issue by default
Date Tue, 10 Nov 2020 22:59:51 GMT
+1 for Apache Spark 3.1.0.


On Tue, Nov 10, 2020 at 6:17 AM Tom Graves <>

> +1 since its a correctness issue, I think its ok to change the behavior to
> make sure the user is aware of it and let them decide.
> Tom
> On Saturday, November 7, 2020, 01:00:11 AM CST, Liang-Chi Hsieh <
>> wrote:
> Hi devs,
> In Spark structured streaming, chained stateful operators possibly produces
> incorrect results under the global watermark. SPARK-33259
> ( has an example
> demostrating what the correctness issue could be.
> Currently we don't prevent users running such queries. Because the possible
> correctness in chained stateful operators in streaming query is not
> straightforward for users. From users perspective, it will possibly be
> considered as a Spark bug like SPARK-33259. It is also possible the worse
> case, users are not aware of the correctness issue and use wrong results.
> IMO, it is better to disable such queries and let users choose to run the
> query if they understand there is such risk, instead of implicitly running
> the query and let users to find out correctness issue by themselves.
> I would like to propose to disable the streaming query with possible
> correctness issue in chained stateful operators. The behavior can be
> controlled by a SQL config, so if users understand the risk and still want
> to run the query, they can disable the check.
> In the PR (, the concern I got
> for now is, this changes current behavior and by default it will break some
> existing streaming queries. But I think it is pretty easy to disable the
> check with the new config. In the PR currently there is no objection but
> suggestion to hear more voices. Please let me know if you have some
> thoughts.
> Thanks.
> Liang-Chi Hsieh
> --
> Sent from:
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message