spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jungtaek Lim (JIRA)" <>
Subject [jira] [Commented] (SPARK-23703) Collapse sequential watermarks
Date Thu, 03 May 2018 23:26:00 GMT


Jungtaek Lim commented on SPARK-23703:

Actually I haven't hear about multiple watermarks on same source, which makes the things
complicated. What I've heard is event-time window with single time field, and watermark for
such field. Do you have/hear actual use cases for this?

> Collapse sequential watermarks 
> -------------------------------
>                 Key: SPARK-23703
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Jose Torres
>            Priority: Major
> When there are two sequential EventTimeWatermark nodes in a query plan, the topmost one
overrides the column tracking metadata from its children, but leaves the nodes themselves
untouched. When there is no intervening stateful operation to consume the watermark, we should
remove the lower node entirely.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message