spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: [Structured Streaming] Two watermarks and StreamingQueryListener
Date Fri, 10 Aug 2018 23:14:58 GMT
Structured Streaming internally maintains one global watermark by taking a
min of the two watermarks. Thats why one gets reported. In Spark 2.4, there
will be the option of choosing max instead of min.

Just curious. Why do you have to two watermarks? Whats the query like.

TD

On Thu, Aug 9, 2018 at 3:15 PM, subramgr <subramanian.girish@gmail.com>
wrote:

> Hi,
>
> We have two *flatMapGroupWithState* in our job and we have two
> *withWatermark*
>
> We are getting the event max time, event time and watermarks from
> *QueryProgressEvent*.
>
> Right now it just returns one *watermark* value.
>
> Are two watermarks maintained by Spark or just one.
> If one which one
> If one watermark is maintained per *Dataframe* how do I get the values for
> them ?
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message