spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jungtaek Lim (Jira)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-28074) [SS] Log warn message on possible correctness issue for multiple stateful operations in single query
Date Tue, 01 Oct 2019 06:58:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-28074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jungtaek Lim updated SPARK-28074:
---------------------------------
    Summary: [SS] Log warn message on possible correctness issue for multiple stateful operations
in single query  (was: [SS] Document caveats on using multiple stateful operations in single
query)

> [SS] Log warn message on possible correctness issue for multiple stateful operations
in single query
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-28074
>                 URL: https://issues.apache.org/jira/browse/SPARK-28074
>             Project: Spark
>          Issue Type: Documentation
>          Components: Documentation, Structured Streaming
>    Affects Versions: 3.0.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Minor
>             Fix For: 3.0.0
>
>
> Please refer [https://lists.apache.org/thread.html/cc6489a19316e7382661d305fabd8c21915e5faf6a928b4869ac2b4a@%3Cdev.spark.apache.org%3E] to
see rationalization of the issue.
> tl;dr. Using multiple stateful operators without fully understanding how global watermark
affects the query brings correctness issue.
> While we may not want to fully block the operations given some users could leverage it
with understanding of global watermark, it would be necessary for end users to warn on such
breaking-correctness possibility, and provide alternative(s).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message