spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject [SS][2.4.4] Confused with "WatermarkTracker: Event time watermark didn't move"?
Date Tue, 08 Oct 2019 09:02:45 GMT
Hi,

I haven't spent much time on it, but the following DEBUG message
from WatermarkTracker sparked my interest :)

I ran a streaming aggregation in Append mode and got the messages:

19/10/08 10:48:56 DEBUG WatermarkTracker: Observed event time stats 0:
EventTimeStats(15000,1000,8000.0,2)
19/10/08 10:48:56 INFO WatermarkTracker: Updating event-time watermark from
0 to 5000 ms
19/10/08 10:48:56 DEBUG WatermarkTracker: Event time watermark didn't move:
5000 < 5000

I think the DEBUG message "Event time watermark didn't move" seems
incorrect given that the query has just started and "Observed event time
stats". It's true that the event-time watermark didn't move if it was 5000
before, but it was not as it has just started from scratch (no checkpointed
state).

Can anyone shed some light on this? I'll be digging deeper in a bit, but am
hoping to get some more info before. Thanks!

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski

Mime
View raw message