spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KevinZwx <>
Subject [Structured Streaming]Usage of watermark
Date Thu, 31 Aug 2017 06:42:47 GMT

I'm a little confused about the usage of watermark in SS. According to the
guideline, when we use a window-based grouping, SS will automatically handle
the late event and we should use watermark to limit the state like
this(specify a watermark before groupBy):

val words = ... // streaming DataFrame of schema { timestamp: Timestamp,
word: String }

// Group the data by window and word and compute the count of each group
val windowedCounts = words
    .withWatermark("timestamp", "10 minutes")
        window($"timestamp", "10 minutes", "5 minutes"),

and when I use dropDuplicates I also need a watermark to limit the state,
like this(specify a watermark before dropDuplicates)

val streamingDf = spark.readStream. ...  // columns: guid, eventTime, ...

// Without watermark using guid column

// With watermark using guid and eventTime columns
  .withWatermark("eventTime", "10 seconds")
  .dropDuplicates("guid", "eventTime")

so if I use a dropDuplicates before a window-based grouping(like below),
should I use two watermarks(one for dropDuplicate and the other for window)? 

val results = events
      window($"timestamp", "1 day"),
      .dropDuplicates("uuid", "window")

Sent from:

To unsubscribe e-mail:

View raw message